2024-03-25T18:32:46.016 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:32:46.016 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/orig.config.yaml 2024-03-25T18:32:46.059 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:32:46.135 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621333' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQnb3SNdRoZlsx73/yAyHFAb4x75q5beKbDacY86ZZwjKwAPqjOPSE/sCU7yAAbXbRFZY8kYDjjaHh33oivqv4= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM1L1qYqDlKANLXcEsGewMeo/VqOsIiqfBeFmUTo5wGc3fWIIJIbguckv6PzY9sD8SbC2cqXjydLyGeYjcWgBQQ= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:32:46.135 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:32:46.136 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:32:46.136 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:32:46.138 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:32:46.163 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:32:46.163 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:32:46.163 INFO:teuthology.packaging:ref: None 2024-03-25T18:32:46.163 INFO:teuthology.packaging:tag: None 2024-03-25T18:32:46.164 INFO:teuthology.packaging:branch: main 2024-03-25T18:32:46.164 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:32:46.164 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:32:46.408 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:32:46.410 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:32:46.416 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:32:46.416 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:32:46.422 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:32:46.438 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:32:46.444 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:32:46.472 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:27:15.929193', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQnb3SNdRoZlsx73/yAyHFAb4x75q5beKbDacY86ZZwjKwAPqjOPSE/sCU7yAAbXbRFZY8kYDjjaHh33oivqv4='} 2024-03-25T18:32:46.493 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:27:15.927280', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM1L1qYqDlKANLXcEsGewMeo/VqOsIiqfBeFmUTo5wGc3fWIIJIbguckv6PzY9sD8SbC2cqXjydLyGeYjcWgBQQ='} 2024-03-25T18:32:46.493 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:32:46.500 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:32:46.500 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:32:46.500 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:32:46.569 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f620cdfa0d0>, signals=[15]) 2024-03-25T18:32:46.570 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:32:46.575 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:32:46.575 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T18:32:46.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:32:46.666 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T18:32:46.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:32:46.755 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:32:46.760 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-03-25T18:32:46.778 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-03-25T18:32:46.778 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-03-25T18:32:46.835 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-03-25T18:32:46.835 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:32:46.836 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:32:46.837 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-03-25T18:32:46.858 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-03-25T18:32:46.875 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-03-25T18:32:46.875 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-03-25T18:32:46.930 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-03-25T18:32:46.930 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="9" 2024-03-25T18:32:46.930 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-03-25T18:32:46.930 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="9" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:32:46.931 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:32:46.931 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-03-25T18:32:46.949 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:32:46.958 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:32:46.962 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:32:46.963 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:32:46.965 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:32:46.985 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:32:46.990 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:32:46.990 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:32:47.021 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:32:47.121 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:32:47.168 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:32:47.188 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:32:47.188 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:32:47.188 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:32:47.189 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:32:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:47.205 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-03-25T18:32:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:47.206 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-03-25T18:32:47.262 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:32:47.263 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-361.el9.x86_64 2024-03-25T18:32:47.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-03-25T18:32:47.264 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:32:47.264 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.14.0-361.el9.x86_64 2024-03-25T18:32:47.264 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2024-03-25T18:32:49.361 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:32:49.381 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T18:32:50.778 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-25T18:32:51.023 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-25T18:32:54.664 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-25T18:32:55.153 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:32:59.661 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 4.8 MB/s | 19 MB 00:03 2024-03-25T18:33:00.280 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-03-25T18:33:07.924 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-25T18:33:08.254 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 1.2 MB/s | 6.1 MB 00:05 2024-03-25T18:33:09.892 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-25T18:33:10.220 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-25T18:33:11.263 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T18:33:12.146 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T18:33:16.193 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-25T18:33:16.987 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-25T18:33:18.325 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:33:18.326 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:33:18.420 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:33:18.427 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:33:18.428 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:33:18.428 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-03-25T18:33:18.428 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:33:18.428 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 76 M 2024-03-25T18:33:18.428 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 123 M 2024-03-25T18:33:18.428 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:33:18.686 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:18.701 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:18.738 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:18.761 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:18.778 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:18.826 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:18.826 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:18.843 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:18.906 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:18.965 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.004 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.037 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.037 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.072 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.072 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.090 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.091 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.094 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:33:19.094 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:33:19.108 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.137 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.154 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.181 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout:Install 4 Packages 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 76 M 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 123 M 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:33:19.217 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.274 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.306 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.339 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.436 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:19.467 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:19.478 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.480 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.480 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.498 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:19.570 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.570 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.580 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.639 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.639 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.644 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.780 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.787 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:19.790 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.816 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:19.817 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:19.823 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:33:19.841 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.844 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.844 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:19.850 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:33:19.854 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.875 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:33:19.883 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.917 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.917 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.924 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.974 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:19.981 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:19.983 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:20.013 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:20.016 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:33:20.044 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:20.047 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:33:20.062 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:33:20.148 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 41 kB/s | 66 kB 00:01 2024-03-25T18:33:20.176 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:20.228 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:20.247 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:20.258 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:20.418 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:20.632 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:20.667 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:20.696 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:20.745 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 6.8 MB/s | 15 MB 00:02 2024-03-25T18:33:20.748 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:20.773 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:20.807 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:20.835 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:20.861 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:20.894 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:20.931 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:20.937 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:20.964 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:20.971 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:20.981 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:21.029 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:21.047 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:21.104 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 13 MB/s | 33 MB 00:02 2024-03-25T18:33:21.104 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:21.113 INFO:teuthology.orchestra.run.smithi178.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 40 kB/s | 66 kB 00:01 2024-03-25T18:33:21.124 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:21.171 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:21.182 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:21.196 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:21.350 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:21.416 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:21.472 INFO:teuthology.orchestra.run.smithi178.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 7.4 MB/s | 15 MB 00:02 2024-03-25T18:33:21.497 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:21.510 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:21.593 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:33:21.673 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:21.691 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:21.768 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:21.907 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:33:22.014 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:22.339 INFO:teuthology.orchestra.run.smithi178.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-25T18:33:22.341 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:33:22.501 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:33:22.582 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:33:22.629 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 11 MB/s | 27 MB 00:02 2024-03-25T18:33:22.635 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:22.635 INFO:teuthology.orchestra.run.smithi110.stdout:Total 18 MB/s | 76 MB 00:04 2024-03-25T18:33:22.638 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:33:23.289 INFO:teuthology.orchestra.run.smithi178.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-25T18:33:23.293 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:23.293 INFO:teuthology.orchestra.run.smithi178.stdout:Total 19 MB/s | 76 MB 00:04 2024-03-25T18:33:23.302 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:33:23.680 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:33:23.680 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:33:23.955 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:33:24.322 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:33:24.322 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:33:24.683 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:33:24.683 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:33:25.318 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:33:25.318 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:33:27.148 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:33:27.776 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:33:27.790 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:33:27.857 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:33:28.563 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:33:28.647 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:33:30.591 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:33:30.718 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:33:31.391 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:33:31.525 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:33:35.553 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:33:35.652 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:33:36.239 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:33:36.275 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:33:40.352 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:33:41.003 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:09.408 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:09.408 INFO:teuthology.orchestra.run.smithi110.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:34:09.408 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:34:09.453 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:09.824 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:34:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:34:09.825 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:34:10.237 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:10.237 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:34:10.237 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:34:10.237 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:34:10.238 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:34:10.238 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:34:10.238 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:34:10.238 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:34:10.238 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:34:10.268 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:10.268 INFO:teuthology.orchestra.run.smithi178.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:34:10.268 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:34:10.314 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:10.585 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:34:10.677 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:34:10.678 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:34:10.678 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:34:10.678 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:34:10.678 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:34:10.678 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:34:10.679 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:34:10.679 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:34:10.703 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:10.703 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:34:10.703 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:34:10.703 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:34:10.758 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:34:10.758 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:34:10.759 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-03-25T18:34:10.801 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-03-25T18:34:10.801 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-03-25T18:34:10.801 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-03-25T18:34:10.801 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 68.el9 2024-03-25T18:34:10.801 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-03-25T18:34:10.801 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:34:10.801 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8264254 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:34:10.802 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:34:10.803 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:34:10.803 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:34:10.803 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-03-25T18:34:10.803 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:34:10.803 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:34:10.804 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:34:10.804 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:34:10.804 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:34:11.151 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:34:11.393 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-03-25T18:34:11.473 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:34:11.581 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:34:11.581 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:34:11.581 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:34:11.581 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:34:11.581 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:34:11.581 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:34:11.582 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:34:11.582 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:34:11.660 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:34:11.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:34:11.661 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.06 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 68.el9 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:Group : Unspecified 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 8264254 2024-03-25T18:34:11.694 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : builder@centos.org 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:34:11.695 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2024-03-25T18:34:11.696 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:34:11.696 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:34:11.696 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:34:11.697 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:34:11.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:34:12.256 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-03-25T18:34:12.753 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:34:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-03-25T18:34:12.803 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-03-25T18:34:12.818 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.rRobYkP6cB 2024-03-25T18:34:12.819 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rRobYkP6cB 2024-03-25T18:34:12.888 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.rRobYkP6cB 2024-03-25T18:34:13.033 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.rRobYkP6cB is 6KB 2024-03-25T18:34:13.053 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.rRobYkP6cB 2024-03-25T18:34:13.067 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:34:13.140 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:34:13.140 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:34:13.140 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:34:13.140 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:34:13.141 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:34:13.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-03-25T18:34:13.628 INFO:teuthology.orchestra.run.smithi178.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:34:13.656 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-03-25T18:34:13.664 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-03-25T18:34:13.679 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.436eY6MhTk 2024-03-25T18:34:13.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.436eY6MhTk 2024-03-25T18:34:13.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.436eY6MhTk 2024-03-25T18:34:13.886 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.436eY6MhTk is 6KB 2024-03-25T18:34:13.906 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.436eY6MhTk 2024-03-25T18:34:13.919 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:34:13.984 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:34:13.984 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:34:13.984 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:34:13.984 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:34:13.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:34:14.112 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-03-25T18:34:43.319 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:34:43.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-25T18:34:43.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:34:44.115 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:34:44.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-25T18:34:44.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:35:01.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-25T18:35:02.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-03-25T18:35:10.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-25T18:35:10.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:35:11.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-25T18:35:11.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:35:14.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-03-25T18:35:26.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-25T18:35:26.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:36:10.847 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:36:22.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-25T18:36:22.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:36:23.274 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-03-25T18:36:23.665 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-25T18:36:23.665 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:36:23.666 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-03-25T18:36:23.682 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:36:23.682 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:36:23.682 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:36:23.682 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:36:24.683 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:36:24.683 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:36:24.764 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-03-25T18:36:24.801 DEBUG:teuthology.parallel:result is None 2024-03-25T18:36:26.638 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:36:41.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-25T18:36:41.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:36:42.080 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-03-25T18:36:42.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-25T18:36:42.470 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:36:42.471 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-03-25T18:36:42.486 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:36:42.486 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:36:42.486 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:36:42.487 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:36:43.487 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:36:43.488 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:36:43.568 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-03-25T18:36:43.604 DEBUG:teuthology.parallel:result is None 2024-03-25T18:36:43.605 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:36:43.612 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:36:43.613 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:36:43.615 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:36:43.633 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:36:43.647 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:36:43.661 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:36:43.661 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:36:43.673 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:36:43.703 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:36:43.714 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:36:43.715 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:36:43.737 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:36:43.771 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:36:43.774 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:36:43.785 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:36:43.788 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:36:43.790 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:36:43.801 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:36:43.801 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:36:43.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:36:43.861 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:36:43.872 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:36:43.872 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:36:43.903 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:36:43.918 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:36:44.000 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:36:44.060 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:36:44.101 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:36:44.143 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T18:36:44.143 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:36:44.210 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:36:44.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:36:44.304 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:36:44.387 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:36:44.431 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T18:36:44.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:36:44.499 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-03-25T18:36:44.501 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-03-25T18:36:44.541 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:36:44.588 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:36:45.032 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:36:45.040 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:36:45.040 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:36:45.140 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:36:45.177 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-03-25T18:36:45.247 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-03-25T18:36:45.249 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-03-25T18:36:45.301 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-03-25T18:36:45.302 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:36:45.302 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-03-25T18:36:45.327 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-03-25T18:36:45.327 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-03-25T18:36:45.356 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-03-25T18:36:45.356 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:36:45.356 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:36:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:36:45.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:36:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:36:45.424 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:36:45.424 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:36:45.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:36:45.500 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:36:45.541 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T18:36:45.547 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T18:36:45.699 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:36:45.718 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:36:45.719 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:41:34.470 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-03-25T18:41:34.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-25T18:41:34.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:41:34.563 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-03-25T18:41:34.595 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-25T18:41:34.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-25T18:41:34.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:41:34.685 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-03-25T18:41:34.765 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-25T18:41:34.765 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:41:34.776 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:41:34.776 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:41:34.777 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:41:34.779 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:41:34.779 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:41:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:41:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:41:34.837 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:41:34.853 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-03-25T18:41:34.854 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:41:34.867 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-03-25T18:41:34.883 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:41:34.883 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-03-25T18:41:34.897 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-03-25T18:41:34.899 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:41:34.913 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:41:34.929 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:41:34.950 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:41:34.954 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:41:34.954 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-03-25T18:41:34.954 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:34.954 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:34.954 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:34.954 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:34.987 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:41:34.991 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:41:34.991 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-03-25T18:41:34.991 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:34.991 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:34.991 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:34.992 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:34.992 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:41:35.000 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:41:35.000 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:41:35.001 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:41:35.001 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:41:35.006 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:41:35.006 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:41:35.007 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:41:35.008 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:41:35.010 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:41:35.010 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:41:35.241 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:41:35.241 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:41:35.247 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:41:35.247 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:41:35.586 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:41:35.586 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T18:41:35.587 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:41:35.593 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:41:35.593 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T18:41:35.594 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:41:35.616 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:41:35.616 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:41:35.622 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:41:35.622 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:41:35.684 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:41:35.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:41:35.763 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:41:35.769 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:41:35.830 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-03-25T18:41:35.831 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-03-25T18:41:35.835 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-03-25T18:41:35.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-03-25T18:41:36.074 INFO:teuthology.orchestra.run.smithi110.stdout:74 files removed 2024-03-25T18:41:36.090 INFO:teuthology.orchestra.run.smithi178.stdout:74 files removed 2024-03-25T18:41:36.101 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw 2024-03-25T18:41:36.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2024-03-25T18:41:37.505 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 75 kB/s | 82 kB 00:01 2024-03-25T18:41:37.538 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 72 kB/s | 82 kB 00:01 2024-03-25T18:41:38.431 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:41:38.489 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T18:41:39.406 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2024-03-25T18:41:39.481 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2024-03-25T18:41:39.908 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:41:40.200 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-25T18:41:41.342 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-25T18:41:41.626 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T18:41:45.455 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:41:45.714 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:41:50.583 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-03-25T18:41:51.008 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-03-25T18:41:57.552 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.1 MB 00:03 2024-03-25T18:41:58.235 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:41:58.908 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-25T18:42:00.051 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-25T18:42:00.507 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T18:42:01.886 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T18:42:05.018 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T18:42:06.876 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T18:42:07.473 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:42:07.483 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:42:07.484 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:42:07.485 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:42:07.486 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout:Installing weak dependencies: 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:42:07.487 INFO:teuthology.orchestra.run.smithi110.stdout:Install 30 Packages 2024-03-25T18:42:07.488 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2024-03-25T18:42:07.488 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:42:07.488 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 58 M 2024-03-25T18:42:07.488 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:42:09.338 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:42:09.349 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:42:09.350 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:42:09.351 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:42:09.352 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:42:09.353 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:42:09.353 INFO:teuthology.orchestra.run.smithi178.stdout:Install 30 Packages 2024-03-25T18:42:09.353 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2024-03-25T18:42:09.353 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:42:09.353 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 58 M 2024-03-25T18:42:09.353 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:42:21.109 INFO:teuthology.orchestra.run.smithi110.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 412 kB/s | 5.3 MB 00:13 2024-03-25T18:42:21.251 INFO:teuthology.orchestra.run.smithi110.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 178 kB/s | 25 kB 00:00 2024-03-25T18:42:23.101 INFO:teuthology.orchestra.run.smithi110.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 401 kB/s | 742 kB 00:01 2024-03-25T18:42:24.459 INFO:teuthology.orchestra.run.smithi110.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 367 kB/s | 498 kB 00:01 2024-03-25T18:42:25.192 INFO:teuthology.orchestra.run.smithi178.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 353 kB/s | 5.3 MB 00:15 2024-03-25T18:42:25.383 INFO:teuthology.orchestra.run.smithi178.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 131 kB/s | 25 kB 00:00 2024-03-25T18:42:27.983 INFO:teuthology.orchestra.run.smithi178.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 285 kB/s | 742 kB 00:02 2024-03-25T18:42:29.048 INFO:teuthology.orchestra.run.smithi110.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 470 kB/s | 9.7 MB 00:21 2024-03-25T18:42:29.075 INFO:teuthology.orchestra.run.smithi178.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 457 kB/s | 498 kB 00:01 2024-03-25T18:42:29.335 INFO:teuthology.orchestra.run.smithi110.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 157 kB/s | 45 kB 00:00 2024-03-25T18:42:29.785 INFO:teuthology.orchestra.run.smithi110.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 292 kB/s | 131 kB 00:00 2024-03-25T18:42:30.210 INFO:teuthology.orchestra.run.smithi110.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 381 kB/s | 162 kB 00:00 2024-03-25T18:42:31.035 INFO:teuthology.orchestra.run.smithi110.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 388 kB/s | 320 kB 00:00 2024-03-25T18:42:31.161 INFO:teuthology.orchestra.run.smithi178.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 464 kB/s | 9.7 MB 00:21 2024-03-25T18:42:31.534 INFO:teuthology.orchestra.run.smithi178.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 121 kB/s | 45 kB 00:00 2024-03-25T18:42:31.735 INFO:teuthology.orchestra.run.smithi110.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 426 kB/s | 298 kB 00:00 2024-03-25T18:42:31.986 INFO:teuthology.orchestra.run.smithi110.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 397 kB/s | 99 kB 00:00 2024-03-25T18:42:32.017 INFO:teuthology.orchestra.run.smithi178.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 272 kB/s | 131 kB 00:00 2024-03-25T18:42:32.111 INFO:teuthology.orchestra.run.smithi110.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 848 kB/s | 106 kB 00:00 2024-03-25T18:42:32.162 INFO:teuthology.orchestra.run.smithi110.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:42:32.203 INFO:teuthology.orchestra.run.smithi110.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T18:42:32.312 INFO:teuthology.orchestra.run.smithi110.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-25T18:42:32.409 INFO:teuthology.orchestra.run.smithi178.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 413 kB/s | 162 kB 00:00 2024-03-25T18:42:32.446 INFO:teuthology.orchestra.run.smithi110.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-25T18:42:32.479 INFO:teuthology.orchestra.run.smithi110.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:42:32.546 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:32.681 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:32.802 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:32.934 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:32.961 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:33.014 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:33.174 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:33.318 INFO:teuthology.orchestra.run.smithi178.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 353 kB/s | 320 kB 00:00 2024-03-25T18:42:33.523 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:33.558 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:42:34.176 INFO:teuthology.orchestra.run.smithi178.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 347 kB/s | 298 kB 00:00 2024-03-25T18:42:34.213 INFO:teuthology.orchestra.run.smithi110.stdout:(18/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 535 kB/s | 5.1 MB 00:09 2024-03-25T18:42:34.248 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:34.270 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.281 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:34.288 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.337 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:34.372 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:34.389 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.435 INFO:teuthology.orchestra.run.smithi178.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 385 kB/s | 99 kB 00:00 2024-03-25T18:42:34.471 INFO:teuthology.orchestra.run.smithi110.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 111 kB/s | 221 kB 00:01 2024-03-25T18:42:34.516 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.528 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:34.543 INFO:teuthology.orchestra.run.smithi178.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T18:42:34.560 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:34.585 INFO:teuthology.orchestra.run.smithi178.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T18:42:34.594 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:34.596 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:34.611 INFO:teuthology.orchestra.run.smithi178.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T18:42:34.626 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:34.644 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.661 INFO:teuthology.orchestra.run.smithi178.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-25T18:42:34.678 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:34.697 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.697 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:42:34.703 INFO:teuthology.orchestra.run.smithi178.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:42:34.751 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.768 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.776 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:34.778 INFO:teuthology.orchestra.run.smithi178.stdout:(17/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 916 kB/s | 5.1 MB 00:05 2024-03-25T18:42:34.804 INFO:teuthology.orchestra.run.smithi178.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 409 kB/s | 41 kB 00:00 2024-03-25T18:42:34.805 INFO:teuthology.orchestra.run.smithi110.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 61 kB/s | 36 kB 00:00 2024-03-25T18:42:34.838 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:42:34.838 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:42:34.856 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:34.858 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.858 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.897 INFO:teuthology.orchestra.run.smithi110.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.3 MB/s | 309 kB 00:00 2024-03-25T18:42:34.897 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:42:34.927 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.937 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.949 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.966 INFO:teuthology.orchestra.run.smithi110.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:34.997 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:35.005 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:35.039 INFO:teuthology.orchestra.run.smithi110.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-03-25T18:42:35.065 INFO:teuthology.orchestra.run.smithi110.stdout:(23/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 83 kB/s | 47 kB 00:00 2024-03-25T18:42:35.090 INFO:teuthology.orchestra.run.smithi110.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-03-25T18:42:35.123 INFO:teuthology.orchestra.run.smithi110.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-25T18:42:35.127 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:35.148 INFO:teuthology.orchestra.run.smithi110.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 579 kB/s | 49 kB 00:00 2024-03-25T18:42:35.151 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:35.190 INFO:teuthology.orchestra.run.smithi110.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T18:42:35.193 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:35.216 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:35.224 INFO:teuthology.orchestra.run.smithi110.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T18:42:35.274 INFO:teuthology.orchestra.run.smithi110.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-25T18:42:35.318 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:35.350 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:35.384 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:35.402 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:35.403 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:35.419 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:35.711 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:42:35.748 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:42:35.786 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:42:35.836 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:42:35.948 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:35.997 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:36.299 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:36.351 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:36.396 INFO:teuthology.orchestra.run.smithi178.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 23 kB/s | 36 kB 00:01 2024-03-25T18:42:36.431 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:42:36.463 INFO:teuthology.orchestra.run.smithi178.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 131 kB/s | 221 kB 00:01 2024-03-25T18:42:36.471 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:36.536 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:36.605 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:36.732 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:42:36.763 INFO:teuthology.orchestra.run.smithi178.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-25T18:42:36.796 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:36.828 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:42:36.930 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:36.949 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:42:37.133 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:42:37.209 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:42:37.397 INFO:teuthology.orchestra.run.smithi178.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-03-25T18:42:37.398 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:37.455 INFO:teuthology.orchestra.run.smithi178.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-03-25T18:42:37.478 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:42:37.514 INFO:teuthology.orchestra.run.smithi178.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-03-25T18:42:37.539 INFO:teuthology.orchestra.run.smithi178.stdout:(25/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 41 kB/s | 47 kB 00:01 2024-03-25T18:42:37.573 INFO:teuthology.orchestra.run.smithi178.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T18:42:37.632 INFO:teuthology.orchestra.run.smithi178.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T18:42:37.807 INFO:teuthology.orchestra.run.smithi178.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-25T18:42:37.866 INFO:teuthology.orchestra.run.smithi178.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-25T18:42:39.066 INFO:teuthology.orchestra.run.smithi110.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 845 kB/s | 3.1 MB 00:03 2024-03-25T18:42:42.591 INFO:teuthology.orchestra.run.smithi110.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 469 kB/s | 3.4 MB 00:07 2024-03-25T18:42:42.932 INFO:teuthology.orchestra.run.smithi178.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 674 kB/s | 3.4 MB 00:05 2024-03-25T18:42:44.215 INFO:teuthology.orchestra.run.smithi178.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 505 kB/s | 3.1 MB 00:06 2024-03-25T18:42:48.524 INFO:teuthology.orchestra.run.smithi110.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 500 kB/s | 20 MB 00:40 2024-03-25T18:42:48.531 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:42:48.531 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.4 MB/s | 58 MB 00:41 2024-03-25T18:42:48.565 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:42:48.603 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:42:48.603 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:42:49.070 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:42:49.070 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:42:50.527 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:42:50.615 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:42:50.645 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:42:50.662 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:42:50.685 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:42:50.702 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:42:50.902 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:42:50.957 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:42:51.316 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:42:51.356 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:42:51.447 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:42:51.534 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:42:51.557 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:42:51.603 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:42:51.632 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:42:51.675 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:42:51.743 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:42:51.768 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:42:51.832 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:42:51.866 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:42:51.974 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:42:52.009 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:42:52.054 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:42:52.090 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:42:52.120 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:42:52.152 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:42:52.469 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:42:52.499 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:42:52.580 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:42:52.612 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:42:52.654 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:42:53.113 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:42:53.143 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:42:53.214 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:42:53.282 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:42:53.313 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:42:55.393 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:42:55.433 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:42:55.991 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:42:56.028 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:42:56.118 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:42:56.118 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:42:56.119 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:42:56.119 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:42:56.154 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:43:00.955 INFO:teuthology.orchestra.run.smithi178.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 397 kB/s | 20 MB 00:51 2024-03-25T18:43:00.963 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:43:00.963 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.1 MB/s | 58 MB 00:51 2024-03-25T18:43:00.993 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:43:01.030 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:43:01.031 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:43:01.519 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:43:01.519 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:43:02.972 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:43:03.052 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:43:03.084 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:43:03.110 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:43:03.133 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:43:03.149 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:43:03.326 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:43:03.380 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:43:03.698 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:43:03.753 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:43:03.836 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:43:03.923 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:43:03.947 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:43:04.010 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:43:04.038 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:43:04.082 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:43:04.150 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:43:04.174 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:43:04.238 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:43:04.265 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:43:04.371 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:43:04.408 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:43:04.449 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:43:04.489 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:43:04.518 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:43:04.551 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:43:04.869 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:43:04.897 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:43:04.987 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:43:05.019 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:43:05.060 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:43:05.370 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:43:05.370 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-03-25T18:43:05.370 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-03-25T18:43:05.370 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-03-25T18:43:05.371 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-03-25T18:43:05.371 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-03-25T18:43:05.371 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-03-25T18:43:05.371 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-03-25T18:43:05.371 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:43:05.509 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:43:05.534 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:43:05.605 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:43:05.673 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:43:05.703 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:43:06.292 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:43:06.317 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:43:06.317 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:43:06.317 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:43:06.317 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:43:06.317 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:43:06.317 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:43:06.319 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:43:06.412 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:43:06.412 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:43:07.533 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:43:07.533 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:43:07.533 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:43:07.533 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:43:07.533 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:43:07.533 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:43:07.533 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:43:07.533 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:43:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:43:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:43:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:43:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:43:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:43:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:43:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:43:07.534 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:43:07.535 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:43:07.536 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:43:07.536 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:43:07.536 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:43:07.536 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:43:07.897 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:43:07.941 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:43:08.485 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:43:08.520 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:43:08.609 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:43:08.609 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:43:08.609 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:43:08.609 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:43:08.628 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:43:08.628 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.629 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:43:08.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:43:08.631 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:43:08.654 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:43:08.966 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test 2024-03-25T18:43:09.519 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:05 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:43:09.668 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:43:09.669 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:43:09.669 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:43:09.669 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:43:09.669 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:43:09.669 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:43:09.669 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:43:09.669 INFO:teuthology.orchestra.run.smithi110.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-03-25T18:43:09.670 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:43:09.671 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 48 M 2024-03-25T18:43:09.671 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 204 M 2024-03-25T18:43:09.671 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:43:09.959 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:43:10.034 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-25T18:43:10.067 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T18:43:10.551 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 212 kB/s | 163 kB 00:00 2024-03-25T18:43:10.576 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 368 kB/s | 187 kB 00:00 2024-03-25T18:43:18.092 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:43:18.093 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-03-25T18:43:18.093 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-03-25T18:43:18.093 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-03-25T18:43:18.093 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-03-25T18:43:18.093 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-03-25T18:43:18.093 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /root 2024-03-25T18:43:18.093 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-03-25T18:43:18.093 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:43:19.035 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:43:19.059 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:43:19.059 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:43:19.059 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:43:19.059 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:43:19.059 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:43:19.059 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:43:19.061 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:43:19.132 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:43:19.132 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:43:20.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:43:20.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:43:20.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:43:21.629 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:43:21.629 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.630 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.631 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:43:21.632 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:43:21.992 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2024-03-25T18:43:22.552 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:43:22.704 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:43:22.705 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:43:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:43:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-03-25T18:43:22.706 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:43:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 48 M 2024-03-25T18:43:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 204 M 2024-03-25T18:43:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:43:22.902 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T18:43:22.960 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T18:43:22.993 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T18:43:23.185 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-25T18:43:23.861 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 146 kB/s | 163 kB 00:01 2024-03-25T18:44:22.869 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 805 kB/s | 47 MB 01:00 2024-03-25T18:44:22.873 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:44:22.874 INFO:teuthology.orchestra.run.smithi178.stdout:Total 820 kB/s | 48 MB 01:00 2024-03-25T18:44:22.883 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:44:22.902 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:44:22.903 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:44:23.283 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:44:23.283 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:44:23.858 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:44:23.935 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:44:23.971 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:44:24.004 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:44:24.049 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:44:24.079 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:44:28.518 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:44:28.595 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:44:30.843 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:44:30.843 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:44:30.843 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:44:30.843 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:44:30.843 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:44:30.843 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:44:31.220 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:44:31.221 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:44:31.537 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2024-03-25T18:44:32.100 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:26 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:44:32.252 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:44:32.255 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================= 2024-03-25T18:44:32.255 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:44:32.255 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================= 2024-03-25T18:44:32.255 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:44:32.255 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:44:32.255 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:44:32.256 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:44:32.257 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout:========================================================================================= 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout:Install 36 Packages 2024-03-25T18:44:32.258 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:44:32.259 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 30 M 2024-03-25T18:44:32.259 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 107 M 2024-03-25T18:44:32.259 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:44:32.870 INFO:teuthology.orchestra.run.smithi178.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-25T18:44:36.028 INFO:teuthology.orchestra.run.smithi178.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 426 kB/s | 1.5 MB 00:03 2024-03-25T18:44:39.203 INFO:teuthology.orchestra.run.smithi178.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 322 kB/s | 2.1 MB 00:06 2024-03-25T18:44:40.111 INFO:teuthology.orchestra.run.smithi178.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 272 kB/s | 247 kB 00:00 2024-03-25T18:44:40.262 INFO:teuthology.orchestra.run.smithi178.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T18:44:40.295 INFO:teuthology.orchestra.run.smithi178.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-25T18:44:40.329 INFO:teuthology.orchestra.run.smithi178.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-03-25T18:44:40.362 INFO:teuthology.orchestra.run.smithi178.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:44:40.396 INFO:teuthology.orchestra.run.smithi178.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:44:40.438 INFO:teuthology.orchestra.run.smithi178.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:44:40.521 INFO:teuthology.orchestra.run.smithi178.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T18:44:40.555 INFO:teuthology.orchestra.run.smithi178.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:44:40.589 INFO:teuthology.orchestra.run.smithi178.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:44:40.622 INFO:teuthology.orchestra.run.smithi178.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T18:44:40.781 INFO:teuthology.orchestra.run.smithi178.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 789 kB/s | 125 kB 00:00 2024-03-25T18:44:40.831 INFO:teuthology.orchestra.run.smithi178.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-03-25T18:44:40.965 INFO:teuthology.orchestra.run.smithi178.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-25T18:44:41.007 INFO:teuthology.orchestra.run.smithi178.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-25T18:44:41.065 INFO:teuthology.orchestra.run.smithi178.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-25T18:44:41.133 INFO:teuthology.orchestra.run.smithi178.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2024-03-25T18:44:41.175 INFO:teuthology.orchestra.run.smithi178.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-25T18:44:41.216 INFO:teuthology.orchestra.run.smithi178.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 427 kB/s | 18 kB 00:00 2024-03-25T18:44:41.259 INFO:teuthology.orchestra.run.smithi178.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 556 kB/s | 23 kB 00:00 2024-03-25T18:44:41.300 INFO:teuthology.orchestra.run.smithi178.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 470 kB/s | 19 kB 00:00 2024-03-25T18:44:41.342 INFO:teuthology.orchestra.run.smithi178.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-03-25T18:44:41.384 INFO:teuthology.orchestra.run.smithi178.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-25T18:44:41.426 INFO:teuthology.orchestra.run.smithi178.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-25T18:44:41.477 INFO:teuthology.orchestra.run.smithi178.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T18:44:41.518 INFO:teuthology.orchestra.run.smithi178.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 396 kB/s | 16 kB 00:00 2024-03-25T18:44:41.560 INFO:teuthology.orchestra.run.smithi178.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-25T18:44:41.594 INFO:teuthology.orchestra.run.smithi178.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2024-03-25T18:44:41.644 INFO:teuthology.orchestra.run.smithi178.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T18:44:41.694 INFO:teuthology.orchestra.run.smithi178.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.4 MB/s | 427 kB 00:00 2024-03-25T18:44:41.736 INFO:teuthology.orchestra.run.smithi178.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 480 kB/s | 20 kB 00:00 2024-03-25T18:44:44.811 INFO:teuthology.orchestra.run.smithi178.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 402 kB/s | 4.7 MB 00:11 2024-03-25T18:45:01.148 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 435 kB/s | 47 MB 01:51 2024-03-25T18:45:01.151 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:01.151 INFO:teuthology.orchestra.run.smithi110.stdout:Total 443 kB/s | 48 MB 01:51 2024-03-25T18:45:01.161 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:45:01.181 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:45:01.181 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:45:01.558 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:45:01.558 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:45:02.152 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:45:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:45:02.273 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:45:02.313 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:45:02.360 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:45:02.385 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:45:06.844 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:45:06.943 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:45:09.261 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:45:09.262 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:45:09.262 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:45:09.262 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:45:09.262 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:45:09.262 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:09.607 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:45:09.900 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph 2024-03-25T18:45:10.469 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:06 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:45:10.625 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:45:10.627 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================= 2024-03-25T18:45:10.627 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:45:10.627 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================= 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:45:10.628 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:45:10.629 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:45:10.630 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================= 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout:Install 36 Packages 2024-03-25T18:45:10.631 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:10.632 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 30 M 2024-03-25T18:45:10.632 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 107 M 2024-03-25T18:45:10.632 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:45:11.261 INFO:teuthology.orchestra.run.smithi110.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T18:45:14.411 INFO:teuthology.orchestra.run.smithi110.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 431 kB/s | 1.5 MB 00:03 2024-03-25T18:45:16.344 INFO:teuthology.orchestra.run.smithi110.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 397 kB/s | 2.1 MB 00:05 2024-03-25T18:45:17.203 INFO:teuthology.orchestra.run.smithi110.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 288 kB/s | 247 kB 00:00 2024-03-25T18:45:17.428 INFO:teuthology.orchestra.run.smithi110.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2024-03-25T18:45:17.487 INFO:teuthology.orchestra.run.smithi110.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T18:45:17.520 INFO:teuthology.orchestra.run.smithi110.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-25T18:45:17.562 INFO:teuthology.orchestra.run.smithi110.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T18:45:17.612 INFO:teuthology.orchestra.run.smithi110.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 775 kB/s | 39 kB 00:00 2024-03-25T18:45:17.705 INFO:teuthology.orchestra.run.smithi110.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2024-03-25T18:45:17.897 INFO:teuthology.orchestra.run.smithi110.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 6.0 MB/s | 1.1 MB 00:00 2024-03-25T18:45:17.930 INFO:teuthology.orchestra.run.smithi110.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:45:17.964 INFO:teuthology.orchestra.run.smithi110.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:45:18.006 INFO:teuthology.orchestra.run.smithi110.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:45:18.073 INFO:teuthology.orchestra.run.smithi110.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-03-25T18:45:18.348 INFO:teuthology.orchestra.run.smithi110.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-03-25T18:45:18.940 INFO:teuthology.orchestra.run.smithi110.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 255 kB/s | 151 kB 00:00 2024-03-25T18:45:19.074 INFO:teuthology.orchestra.run.smithi110.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 326 kB/s | 43 kB 00:00 2024-03-25T18:45:19.241 INFO:teuthology.orchestra.run.smithi110.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-25T18:45:19.458 INFO:teuthology.orchestra.run.smithi110.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-03-25T18:45:19.559 INFO:teuthology.orchestra.run.smithi110.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T18:45:19.667 INFO:teuthology.orchestra.run.smithi110.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-25T18:45:19.784 INFO:teuthology.orchestra.run.smithi110.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 199 kB/s | 23 kB 00:00 2024-03-25T18:45:19.884 INFO:teuthology.orchestra.run.smithi110.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-25T18:45:19.985 INFO:teuthology.orchestra.run.smithi110.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 197 kB/s | 20 kB 00:00 2024-03-25T18:45:20.085 INFO:teuthology.orchestra.run.smithi110.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T18:45:20.218 INFO:teuthology.orchestra.run.smithi110.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 591 kB/s | 79 kB 00:00 2024-03-25T18:45:20.360 INFO:teuthology.orchestra.run.smithi110.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-25T18:45:20.469 INFO:teuthology.orchestra.run.smithi110.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-25T18:45:20.586 INFO:teuthology.orchestra.run.smithi110.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 771 kB/s | 90 kB 00:00 2024-03-25T18:45:20.694 INFO:teuthology.orchestra.run.smithi110.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2024-03-25T18:45:20.828 INFO:teuthology.orchestra.run.smithi110.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-25T18:45:20.978 INFO:teuthology.orchestra.run.smithi110.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-03-25T18:45:21.087 INFO:teuthology.orchestra.run.smithi110.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-25T18:45:22.471 INFO:teuthology.orchestra.run.smithi110.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 428 kB/s | 4.7 MB 00:11 2024-03-25T18:45:23.214 INFO:teuthology.orchestra.run.smithi178.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 364 kB/s | 17 MB 00:47 2024-03-25T18:45:23.217 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:23.217 INFO:teuthology.orchestra.run.smithi178.stdout:Total 596 kB/s | 30 MB 00:50 2024-03-25T18:45:23.241 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:45:23.282 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:45:23.283 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:45:23.697 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:45:23.697 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:45:25.011 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:45:25.074 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:45:25.118 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:45:25.143 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:45:25.215 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:45:25.250 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:45:25.283 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:45:25.382 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:45:25.479 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:45:25.519 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:45:25.592 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:45:25.644 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:45:25.683 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:45:25.740 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:45:25.796 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:45:25.980 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:45:26.029 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:45:26.076 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:45:26.127 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:45:26.157 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:45:26.232 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:45:26.268 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:45:26.301 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:45:26.341 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:45:26.377 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:45:26.411 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:45:26.500 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:45:26.542 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:45:26.613 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:45:26.760 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:45:26.886 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:45:27.041 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:45:27.071 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:45:27.096 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:45:27.096 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:27.096 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:45:27.096 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:45:27.096 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:45:27.096 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:28.530 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:45:28.556 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:45:28.557 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:28.557 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:45:28.557 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:45:28.557 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:45:28.557 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:28.998 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:45:29.021 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:45:29.021 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:29.021 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:45:29.021 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:45:29.021 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:45:29.021 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:29.223 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:45:29.246 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:45:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:45:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:45:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:45:29.246 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:29.312 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:45:31.054 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:45:31.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:45:31.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:45:31.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:45:31.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:45:31.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:45:31.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:45:31.054 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:45:31.055 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:45:31.056 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:45:31.942 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:45:31.942 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:31.942 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:45:31.942 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:45:31.943 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:45:31.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:31.945 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:45:32.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2024-03-25T18:45:32.848 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:26 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:45:32.945 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:45:32.996 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:45:32.997 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:45:32.997 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:45:33.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2024-03-25T18:45:33.618 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:27 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:45:33.766 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:45:33.767 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:45:33.767 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:45:33.767 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:45:33.767 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:45:33.767 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:45:33.767 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:33.767 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:45:33.768 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:45:33.768 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T18:45:33.768 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:33.768 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 747 k 2024-03-25T18:45:33.768 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 754 k 2024-03-25T18:45:33.768 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:45:36.914 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 238 kB/s | 747 kB 00:03 2024-03-25T18:45:36.914 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:36.915 INFO:teuthology.orchestra.run.smithi178.stdout:Total 237 kB/s | 747 kB 00:03 2024-03-25T18:45:36.915 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:45:36.921 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:45:36.921 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:45:36.927 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:45:36.928 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:45:37.002 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:45:37.501 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:45:37.620 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:45:37.972 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:45:38.262 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:45:38.263 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:38.263 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:45:38.263 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:45:38.263 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:38.263 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:45:38.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:45:39.042 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:33 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:45:39.193 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 145 k 2024-03-25T18:45:39.194 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 436 k 2024-03-25T18:45:39.195 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:45:40.477 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 113 kB/s | 145 kB 00:01 2024-03-25T18:45:40.478 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:40.478 INFO:teuthology.orchestra.run.smithi178.stdout:Total 113 kB/s | 145 kB 00:01 2024-03-25T18:45:40.478 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:45:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:45:40.488 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:45:40.543 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:45:40.544 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:45:40.690 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:45:40.753 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:45:40.777 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:45:40.778 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:40.778 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:45:40.778 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:41.775 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:45:41.775 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:41.775 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:45:41.775 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:41.775 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:41.775 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:45:42.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2024-03-25T18:45:42.607 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:36 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:45:42.703 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:45:42.756 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:45:42.757 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:45:42.757 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:45:42.815 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:45:43.374 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:37 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:45:43.524 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:45:43.525 INFO:teuthology.orchestra.run.smithi178.stdout:====================================================================================== 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout:====================================================================================== 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:45:43.526 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:45:43.527 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:45:43.527 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:45:43.527 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:45:43.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:43.527 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:45:43.527 INFO:teuthology.orchestra.run.smithi178.stdout:====================================================================================== 2024-03-25T18:45:43.527 INFO:teuthology.orchestra.run.smithi178.stdout:Install 12 Packages 2024-03-25T18:45:43.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:43.528 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 6.9 M 2024-03-25T18:45:43.528 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 26 M 2024-03-25T18:45:43.528 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:45:44.330 INFO:teuthology.orchestra.run.smithi178.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-25T18:45:44.489 INFO:teuthology.orchestra.run.smithi178.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-25T18:45:44.514 INFO:teuthology.orchestra.run.smithi178.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 48 kB/s | 31 kB 00:00 2024-03-25T18:45:44.539 INFO:teuthology.orchestra.run.smithi178.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.6 MB/s | 238 kB 00:00 2024-03-25T18:45:44.640 INFO:teuthology.orchestra.run.smithi178.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.4 MB/s | 548 kB 00:00 2024-03-25T18:45:44.673 INFO:teuthology.orchestra.run.smithi178.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 586 kB/s | 19 kB 00:00 2024-03-25T18:45:44.798 INFO:teuthology.orchestra.run.smithi178.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-03-25T18:45:44.840 INFO:teuthology.orchestra.run.smithi178.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-03-25T18:45:44.883 INFO:teuthology.orchestra.run.smithi178.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 736 kB/s | 31 kB 00:00 2024-03-25T18:45:44.916 INFO:teuthology.orchestra.run.smithi178.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.1 MB/s | 863 kB 00:00 2024-03-25T18:45:44.942 INFO:teuthology.orchestra.run.smithi178.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-03-25T18:45:45.677 INFO:teuthology.orchestra.run.smithi110.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 549 kB/s | 17 MB 00:31 2024-03-25T18:45:45.682 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:45.682 INFO:teuthology.orchestra.run.smithi110.stdout:Total 867 kB/s | 30 MB 00:35 2024-03-25T18:45:45.707 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:45:45.749 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:45:45.749 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:45:46.166 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:45:46.166 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:45:47.502 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:45:47.563 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:45:47.598 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:45:47.632 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:45:47.702 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:45:47.739 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:45:47.772 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:45:47.864 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:45:47.942 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:45:47.983 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:45:48.055 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:45:48.108 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:45:48.155 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:45:48.205 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:45:48.260 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:45:48.446 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:45:48.502 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:45:48.532 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:45:48.567 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:45:48.597 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:45:48.673 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:45:48.707 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:45:48.740 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:45:48.781 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:45:48.816 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:45:48.850 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:45:48.966 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:45:49.006 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:45:49.079 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:45:49.224 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:45:49.350 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:45:49.516 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:45:49.545 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:45:49.570 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:45:49.571 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:49.571 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:45:49.571 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:45:49.571 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:45:49.571 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:50.033 INFO:teuthology.orchestra.run.smithi178.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 295 kB/s | 1.8 MB 00:06 2024-03-25T18:45:50.036 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:50.037 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T18:45:50.057 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:45:50.076 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:45:50.076 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:45:50.257 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:45:50.257 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:45:50.771 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:45:50.850 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:45:50.893 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:45:50.940 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:45:50.978 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:45:51.021 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:45:51.048 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:45:51.048 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:51.048 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:45:51.048 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:45:51.048 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:45:51.048 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:51.073 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:45:51.195 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:45:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:45:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:45:51.362 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:45:51.393 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:45:51.488 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:45:51.512 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:45:51.512 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:51.512 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:45:51.512 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:45:51.512 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:45:51.512 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:51.635 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:45:51.660 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:45:51.723 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:45:51.745 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:45:51.745 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:45:51.746 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:45:51.746 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:45:51.746 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:45:51.746 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:51.812 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:45:52.534 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:45:52.534 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:45:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:45:52.924 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:45:52.925 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:45:52.925 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:45:52.925 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:45:52.925 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:52.925 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:45:53.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:45:53.442 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:45:53.442 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:45:53.442 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:45:53.442 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:45:53.442 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:45:53.442 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:45:53.443 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:45:53.444 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:45:53.444 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:45:53.444 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:45:53.444 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:45:53.444 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:45:53.444 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:45:53.444 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:45:53.445 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:45:53.445 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:45:53.445 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:45:53.445 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:45:53.445 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:45:53.445 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:45:53.445 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:45:53.445 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:45:53.764 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:03:47 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:45:53.918 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:45:53.920 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout:Install 12 Packages 2024-03-25T18:45:53.921 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:45:53.922 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 37 M 2024-03-25T18:45:53.922 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 209 M 2024-03-25T18:45:53.922 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:45:54.355 INFO:teuthology.orchestra.run.smithi178.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 267 kB/s | 32 kB 00:00 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:45:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:45:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:45:54.396 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:45:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:45:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:45:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:45:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:54.398 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:45:54.414 INFO:teuthology.orchestra.run.smithi178.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 492 kB/s | 17 kB 00:00 2024-03-25T18:45:54.448 INFO:teuthology.orchestra.run.smithi178.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:45:54.623 INFO:teuthology.orchestra.run.smithi178.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T18:45:54.723 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base 2024-03-25T18:45:54.756 INFO:teuthology.orchestra.run.smithi178.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:45:54.824 INFO:teuthology.orchestra.run.smithi178.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T18:45:54.849 INFO:teuthology.orchestra.run.smithi178.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-25T18:45:54.999 INFO:teuthology.orchestra.run.smithi178.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.3 MB/s | 803 kB 00:00 2024-03-25T18:45:55.183 INFO:teuthology.orchestra.run.smithi178.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-25T18:45:55.216 INFO:teuthology.orchestra.run.smithi178.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 910 kB/s | 195 kB 00:00 2024-03-25T18:45:55.315 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:51 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:45:55.408 INFO:teuthology.orchestra.run.smithi178.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-25T18:45:55.412 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:45:55.464 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:45:55.464 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:45:55.465 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:45:55.533 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install cephadm 2024-03-25T18:45:56.091 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:52 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:45:56.239 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:45:56.240 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 747 k 2024-03-25T18:45:56.241 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 754 k 2024-03-25T18:45:56.241 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:45:59.651 INFO:teuthology.orchestra.run.smithi110.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 219 kB/s | 747 kB 00:03 2024-03-25T18:45:59.651 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:59.652 INFO:teuthology.orchestra.run.smithi110.stdout:Total 219 kB/s | 747 kB 00:03 2024-03-25T18:45:59.652 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:45:59.658 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:45:59.658 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:45:59.664 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:45:59.664 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:45:59.744 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:00.209 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:46:00.296 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:46:00.674 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:46:00.961 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:46:00.961 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:00.961 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:00.961 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:00.961 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:00.961 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:01.163 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:46:01.732 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:57 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:01.884 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 145 k 2024-03-25T18:46:01.885 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 436 k 2024-03-25T18:46:01.886 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:46:03.191 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 111 kB/s | 145 kB 00:01 2024-03-25T18:46:03.192 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:03.192 INFO:teuthology.orchestra.run.smithi110.stdout:Total 111 kB/s | 145 kB 00:01 2024-03-25T18:46:03.193 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:46:03.202 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:46:03.202 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:46:03.259 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:46:03.259 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:46:03.409 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:03.492 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:46:03.516 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:46:03.516 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:46:03.516 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:46:03.516 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:04.440 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:46:04.440 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:04.441 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:04.441 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:46:04.441 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:04.441 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:04.728 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr 2024-03-25T18:46:05.303 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:01 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:05.401 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:05.453 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:05.454 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:46:05.454 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:05.525 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:46:06.085 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:02 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:06.232 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:06.233 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================== 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================== 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:46:06.234 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:46:06.235 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:46:06.235 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:46:06.235 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:46:06.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:06.235 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:46:06.235 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================== 2024-03-25T18:46:06.235 INFO:teuthology.orchestra.run.smithi110.stdout:Install 12 Packages 2024-03-25T18:46:06.235 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:06.236 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 6.9 M 2024-03-25T18:46:06.236 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2024-03-25T18:46:06.236 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:46:06.821 INFO:teuthology.orchestra.run.smithi110.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-25T18:46:06.963 INFO:teuthology.orchestra.run.smithi110.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T18:46:06.988 INFO:teuthology.orchestra.run.smithi110.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 49 kB/s | 31 kB 00:00 2024-03-25T18:46:07.005 INFO:teuthology.orchestra.run.smithi110.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.5 MB/s | 238 kB 00:00 2024-03-25T18:46:07.222 INFO:teuthology.orchestra.run.smithi110.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-25T18:46:07.481 INFO:teuthology.orchestra.run.smithi110.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-25T18:46:07.657 INFO:teuthology.orchestra.run.smithi110.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 842 kB/s | 548 kB 00:00 2024-03-25T18:46:07.773 INFO:teuthology.orchestra.run.smithi110.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-25T18:46:07.873 INFO:teuthology.orchestra.run.smithi110.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-25T18:46:07.990 INFO:teuthology.orchestra.run.smithi110.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-25T18:46:08.107 INFO:teuthology.orchestra.run.smithi110.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-25T18:46:12.124 INFO:teuthology.orchestra.run.smithi110.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 316 kB/s | 1.8 MB 00:05 2024-03-25T18:46:12.127 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:12.127 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.2 MB/s | 6.9 MB 00:05 2024-03-25T18:46:12.147 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:46:12.166 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:46:12.166 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:46:12.264 INFO:teuthology.orchestra.run.smithi178.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 420 kB/s | 7.4 MB 00:18 2024-03-25T18:46:12.267 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:12.267 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.0 MB/s | 37 MB 00:18 2024-03-25T18:46:12.311 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:46:12.335 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:46:12.335 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:46:12.336 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:46:12.336 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:46:12.697 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:46:12.698 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:46:12.882 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:12.958 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:46:12.993 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:46:13.039 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:46:13.070 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:46:13.172 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:46:13.304 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:46:13.398 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:46:13.408 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:46:13.442 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:46:13.471 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:46:13.486 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:46:13.502 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:46:13.526 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:46:13.619 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:46:13.663 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:46:13.744 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:46:13.848 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:46:13.870 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:46:13.947 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:46:14.602 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:46:14.608 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:46:14.608 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:46:14.608 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:46:14.609 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:46:14.715 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:46:14.952 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:46:14.953 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:14.953 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:14.953 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:46:14.953 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:14.953 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:14.953 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:14.954 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:46:14.954 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:46:14.954 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:46:14.954 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:46:14.954 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:46:14.954 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:46:14.955 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:46:14.955 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:46:14.955 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:14.955 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:15.189 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:46:15.744 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:15.895 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================================ 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================================ 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:46:15.896 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================================ 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout:Install 12 Packages 2024-03-25T18:46:15.897 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:15.898 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 37 M 2024-03-25T18:46:15.898 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 209 M 2024-03-25T18:46:15.898 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:46:15.946 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:46:16.182 INFO:teuthology.orchestra.run.smithi110.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 235 kB/s | 32 kB 00:00 2024-03-25T18:46:16.240 INFO:teuthology.orchestra.run.smithi110.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 493 kB/s | 17 kB 00:00 2024-03-25T18:46:16.274 INFO:teuthology.orchestra.run.smithi110.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:46:16.441 INFO:teuthology.orchestra.run.smithi110.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-25T18:46:16.574 INFO:teuthology.orchestra.run.smithi110.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:46:16.633 INFO:teuthology.orchestra.run.smithi110.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T18:46:16.658 INFO:teuthology.orchestra.run.smithi110.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T18:46:16.976 INFO:teuthology.orchestra.run.smithi110.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-25T18:46:16.991 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:46:17.009 INFO:teuthology.orchestra.run.smithi110.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-25T18:46:17.027 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:46:17.093 INFO:teuthology.orchestra.run.smithi110.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-03-25T18:46:17.117 INFO:teuthology.orchestra.run.smithi110.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-25T18:46:19.288 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:46:19.288 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:46:19.288 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:46:19.288 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:46:19.288 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:46:19.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:46:19.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:46:19.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:46:19.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:46:19.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:46:19.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:46:19.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:46:19.756 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:46:19.756 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:19.757 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:20.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:46:20.689 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:20.838 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:20.839 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T18:46:20.839 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:46:20.839 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:46:20.840 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:46:20.841 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:20.841 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:46:20.841 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T18:46:20.841 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-03-25T18:46:20.841 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:20.841 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 2.0 M 2024-03-25T18:46:20.841 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 26 M 2024-03-25T18:46:20.842 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:46:21.004 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 205 kB/s | 24 kB 00:00 2024-03-25T18:46:21.046 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2024-03-25T18:46:21.088 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T18:46:21.113 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:46:21.139 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:46:21.156 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T18:46:21.306 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 100 kB/s | 15 kB 00:00 2024-03-25T18:46:21.331 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 168 kB/s | 32 kB 00:00 2024-03-25T18:46:21.440 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.5 MB/s | 203 kB 00:00 2024-03-25T18:46:21.490 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-25T18:46:21.532 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.1 MB/s | 1.0 MB 00:00 2024-03-25T18:46:21.557 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 78 kB/s | 52 kB 00:00 2024-03-25T18:46:21.582 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 974 kB/s | 90 kB 00:00 2024-03-25T18:46:21.586 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:21.586 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-25T18:46:21.586 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:46:21.599 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:46:21.599 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:46:21.689 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:46:21.694 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:46:22.211 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:46:22.379 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:46:22.433 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:46:22.489 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:46:22.526 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:46:22.567 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:46:22.670 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:46:22.788 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:46:22.844 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:46:23.568 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:46:23.625 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:46:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:46:23.708 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:46:23.736 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:46:24.572 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:46:24.572 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:46:24.572 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:46:24.572 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:46:24.572 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:46:24.572 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:46:24.573 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:46:24.573 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:46:24.573 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:46:24.573 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:46:24.573 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:46:24.573 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:46:24.573 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:46:24.954 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:46:24.954 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:24.954 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:46:24.954 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:24.954 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:46:24.954 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:46:24.954 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:46:24.954 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:24.955 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:25.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:46:25.781 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:25.932 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:46:25.933 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:46:25.934 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:46:25.934 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:25.934 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:46:25.934 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T18:46:25.934 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-03-25T18:46:25.934 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:25.934 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 6.8 M 2024-03-25T18:46:25.935 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 33 M 2024-03-25T18:46:25.935 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:46:26.732 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-25T18:46:26.925 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 191 kB/s | 138 kB 00:00 2024-03-25T18:46:26.951 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 332 kB/s | 248 kB 00:00 2024-03-25T18:46:27.226 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 192 kB/s | 58 kB 00:00 2024-03-25T18:46:27.243 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2024-03-25T18:46:27.576 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 650 kB/s | 548 kB 00:00 2024-03-25T18:46:27.577 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:27.578 INFO:teuthology.orchestra.run.smithi178.stdout:Total 4.1 MB/s | 6.8 MB 00:01 2024-03-25T18:46:27.592 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:46:27.604 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:46:27.604 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:46:27.696 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:46:27.696 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:46:27.998 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:46:28.157 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:46:28.250 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:46:28.763 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:46:28.840 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:46:28.901 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:46:28.925 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:46:29.500 INFO:teuthology.orchestra.run.smithi110.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 562 kB/s | 7.4 MB 00:13 2024-03-25T18:46:29.503 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:29.503 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.7 MB/s | 37 MB 00:13 2024-03-25T18:46:29.545 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:46:29.571 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:46:29.571 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:46:29.719 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:46:29.719 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:46:29.719 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:46:29.719 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:46:29.719 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:46:29.719 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:46:29.931 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:46:29.931 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:46:30.033 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:46:30.033 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:30.034 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:46:30.034 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:30.034 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:46:30.034 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:46:30.034 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:46:30.034 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:46:30.035 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:46:30.035 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:30.035 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:30.299 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2024-03-25T18:46:30.631 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:30.690 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:46:30.729 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:46:30.830 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:46:30.865 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:30.866 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:46:31.014 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:31.015 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:31.015 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:46:31.015 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:31.015 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:46:31.015 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 964 k 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.8 M 2024-03-25T18:46:31.016 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:46:31.054 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:46:31.082 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:46:31.129 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 697 kB/s | 79 kB 00:00 2024-03-25T18:46:31.158 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:46:31.798 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:46:31.910 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:46:33.143 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:46:34.037 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 293 kB/s | 885 kB 00:03 2024-03-25T18:46:34.039 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:34.039 INFO:teuthology.orchestra.run.smithi178.stdout:Total 319 kB/s | 964 kB 00:03 2024-03-25T18:46:34.039 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:46:34.063 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:46:34.063 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:46:34.145 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:46:34.146 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:46:34.163 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:46:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:46:34.463 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:46:34.538 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:46:35.239 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:46:35.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:46:35.567 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:46:35.567 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:35.567 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:46:35.567 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:46:35.567 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:35.567 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:35.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-volume 2024-03-25T18:46:36.413 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:30 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:36.565 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:36.566 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:36.566 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-25T18:46:36.566 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:36.566 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:46:36.566 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 332 k 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.5 M 2024-03-25T18:46:36.567 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:46:36.691 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 555 kB/s | 69 kB 00:00 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:46:36.865 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:46:36.866 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:46:36.866 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:46:36.866 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:46:37.251 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:46:37.251 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:37.251 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:37.251 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:37.251 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:37.252 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:37.651 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:46:37.867 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 203 kB/s | 264 kB 00:01 2024-03-25T18:46:37.868 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:37.868 INFO:teuthology.orchestra.run.smithi178.stdout:Total 255 kB/s | 332 kB 00:01 2024-03-25T18:46:37.869 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:46:37.879 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:46:37.879 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:46:37.924 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:46:37.924 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:46:38.072 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:46:38.217 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:38.280 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:46:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:46:38.326 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:46:38.326 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:46:38.326 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:46:38.326 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:38.365 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:38.366 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T18:46:38.366 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:46:38.366 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:46:38.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:46:38.368 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:46:38.368 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:46:38.368 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:46:38.368 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:38.368 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:46:38.368 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T18:46:38.368 INFO:teuthology.orchestra.run.smithi110.stdout:Install 13 Packages 2024-03-25T18:46:38.368 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:38.369 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2024-03-25T18:46:38.369 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2024-03-25T18:46:38.369 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:46:38.526 INFO:teuthology.orchestra.run.smithi110.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 216 kB/s | 24 kB 00:00 2024-03-25T18:46:38.576 INFO:teuthology.orchestra.run.smithi110.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-03-25T18:46:38.626 INFO:teuthology.orchestra.run.smithi110.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:46:38.652 INFO:teuthology.orchestra.run.smithi110.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:46:38.677 INFO:teuthology.orchestra.run.smithi110.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:46:38.702 INFO:teuthology.orchestra.run.smithi110.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2024-03-25T18:46:38.803 INFO:teuthology.orchestra.run.smithi110.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 150 kB/s | 15 kB 00:00 2024-03-25T18:46:38.830 INFO:teuthology.orchestra.run.smithi110.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 211 kB/s | 32 kB 00:00 2024-03-25T18:46:38.930 INFO:teuthology.orchestra.run.smithi110.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.6 MB/s | 203 kB 00:00 2024-03-25T18:46:38.997 INFO:teuthology.orchestra.run.smithi110.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 889 kB/s | 59 kB 00:00 2024-03-25T18:46:39.056 INFO:teuthology.orchestra.run.smithi110.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-03-25T18:46:39.089 INFO:teuthology.orchestra.run.smithi110.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 976 kB/s | 90 kB 00:00 2024-03-25T18:46:39.119 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:46:39.290 INFO:teuthology.orchestra.run.smithi110.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 60 kB/s | 52 kB 00:00 2024-03-25T18:46:39.291 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:39.292 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T18:46:39.292 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:46:39.306 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:46:39.306 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:46:39.400 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:46:39.411 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:46:39.442 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:46:39.442 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:39.442 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:46:39.442 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:39.443 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:46:39.443 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:39.443 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:39.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2024-03-25T18:46:39.921 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:40.093 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:46:40.149 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:46:40.212 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:46:40.242 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:46:40.283 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:46:40.335 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:40.376 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:46:40.477 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:46:40.486 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T18:46:40.487 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:40.488 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 127 k 2024-03-25T18:46:40.488 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 456 k 2024-03-25T18:46:40.488 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:46:40.526 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:46:41.247 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:46:41.291 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:46:41.324 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:46:41.373 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:46:41.395 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:46:41.764 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 99 kB/s | 127 kB 00:01 2024-03-25T18:46:41.764 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:41.765 INFO:teuthology.orchestra.run.smithi178.stdout:Total 99 kB/s | 127 kB 00:01 2024-03-25T18:46:41.765 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:46:41.771 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:46:41.771 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:46:41.833 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:46:41.833 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:46:42.005 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:46:42.104 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:42.156 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:46:42.156 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:46:42.156 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:46:42.156 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:46:42.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:46:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:46:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:46:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:46:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:42.510 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:42.577 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:42.746 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:46:42.848 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:42.849 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:42.849 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:46:42.849 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:46:42.849 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:42.849 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:43.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2024-03-25T18:46:43.304 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:43.454 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:43.454 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:46:43.455 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T18:46:43.456 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-03-25T18:46:43.456 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:43.456 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 6.8 M 2024-03-25T18:46:43.456 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 33 M 2024-03-25T18:46:43.456 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:46:43.631 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:37 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:43.730 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:43.785 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:43.786 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:46:43.786 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:43.845 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2024-03-25T18:46:44.135 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T18:46:44.159 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 871 kB/s | 248 kB 00:00 2024-03-25T18:46:44.410 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:44.460 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 192 kB/s | 58 kB 00:00 2024-03-25T18:46:44.558 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:44.558 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 31 k 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 140 k 2024-03-25T18:46:44.559 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:46:44.585 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T18:46:44.610 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 340 kB/s | 51 kB 00:00 2024-03-25T18:46:44.852 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 141 kB/s | 138 kB 00:00 2024-03-25T18:46:44.855 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:44.855 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-03-25T18:46:44.867 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:46:44.877 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:46:44.878 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:46:44.965 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:46:44.966 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:46:45.183 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 50 kB/s | 31 kB 00:00 2024-03-25T18:46:45.184 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:45.184 INFO:teuthology.orchestra.run.smithi178.stdout:Total 50 kB/s | 31 kB 00:00 2024-03-25T18:46:45.184 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:46:45.187 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:46:45.187 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:46:45.209 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:46:45.209 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:46:45.260 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:45.337 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:46:45.447 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:45.452 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:46:45.495 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:46:45.875 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:46.011 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:46:46.097 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:46:46.156 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:46.156 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:46.156 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:46:46.156 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:46:46.156 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:46.156 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:46.159 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:46:46.183 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:46:46.309 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2024-03-25T18:46:46.872 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:46.951 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:46:46.951 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:46:46.951 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:46:46.951 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:46:46.951 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:46:46.951 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:46:46.969 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:47.021 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:47.021 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:46:47.021 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:47.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2024-03-25T18:46:47.272 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:46:47.272 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:47.272 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:47.272 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:47.272 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:46:47.273 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:46:47.273 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:46:47.273 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:46:47.273 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:46:47.273 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:47.273 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:47.501 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse 2024-03-25T18:46:47.639 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:47.743 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:47.799 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:47.799 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:46:47.800 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:47.848 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2024-03-25T18:46:48.060 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:48.209 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:48.209 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:48.209 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-03-25T18:46:48.210 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:48.211 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 964 k 2024-03-25T18:46:48.211 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.8 M 2024-03-25T18:46:48.211 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:46:48.330 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 662 kB/s | 79 kB 00:00 2024-03-25T18:46:48.429 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:48.529 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:48.581 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:48.581 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:46:48.582 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:48.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2024-03-25T18:46:49.194 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:49.292 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:49.343 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:49.344 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:46:49.344 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:49.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2024-03-25T18:46:49.949 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:50.005 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 493 kB/s | 885 kB 00:01 2024-03-25T18:46:50.007 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:50.007 INFO:teuthology.orchestra.run.smithi110.stdout:Total 536 kB/s | 964 kB 00:01 2024-03-25T18:46:50.008 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:46:50.031 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:46:50.031 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:46:50.045 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:50.096 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:50.097 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:46:50.097 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:50.106 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:46:50.107 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:46:50.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2024-03-25T18:46:50.283 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:50.422 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:46:50.488 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:46:50.705 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:50.802 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:50.853 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:50.854 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T18:46:50.854 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:50.904 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2024-03-25T18:46:51.180 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:46:51.180 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:46:51.468 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:51.497 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:46:51.497 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:51.497 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:51.497 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:46:51.497 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:51.497 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:51.616 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 85 k 2024-03-25T18:46:51.617 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 231 k 2024-03-25T18:46:51.618 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:46:51.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-volume 2024-03-25T18:46:52.236 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 138 kB/s | 85 kB 00:00 2024-03-25T18:46:52.237 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:52.237 INFO:teuthology.orchestra.run.smithi178.stdout:Total 138 kB/s | 85 kB 00:00 2024-03-25T18:46:52.237 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:46:52.245 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:46:52.245 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:46:52.301 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:46:52.301 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:46:52.348 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:52.435 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:46:52.496 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:52.497 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:52.497 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-03-25T18:46:52.497 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:52.497 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:46:52.497 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:46:52.497 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:46:52.498 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:46:52.498 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:52.498 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:46:52.498 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:52.498 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-03-25T18:46:52.498 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:52.498 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 332 k 2024-03-25T18:46:52.499 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.5 M 2024-03-25T18:46:52.499 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:46:52.535 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:52.602 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 666 kB/s | 69 kB 00:00 2024-03-25T18:46:52.946 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:53.250 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:53.251 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:46:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:46:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:53.251 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:46:53.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2024-03-25T18:46:53.810 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 201 kB/s | 264 kB 00:01 2024-03-25T18:46:53.812 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:53.812 INFO:teuthology.orchestra.run.smithi110.stdout:Total 253 kB/s | 332 kB 00:01 2024-03-25T18:46:53.813 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:46:53.822 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:46:53.822 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:46:53.870 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:46:53.870 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:46:54.022 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:54.053 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:46:54.207 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:46:54.207 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:54.207 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T18:46:54.207 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:54.208 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:46:54.208 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:46:54.208 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:54.208 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:46:54.208 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:46:54.208 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T18:46:54.208 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:46:54.208 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.1 M 2024-03-25T18:46:54.209 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 13 M 2024-03-25T18:46:54.209 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:46:54.210 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:46:54.247 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:46:54.263 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:46:54.263 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:46:54.263 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:46:54.263 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:55.146 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:46:55.472 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:46:55.472 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:55.472 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:55.472 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:46:55.473 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:46:55.473 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:55.473 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:55.811 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel 2024-03-25T18:46:56.386 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:56.534 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:56.534 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T18:46:56.535 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:56.536 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 127 k 2024-03-25T18:46:56.536 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 456 k 2024-03-25T18:46:56.536 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:46:57.292 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 167 kB/s | 127 kB 00:00 2024-03-25T18:46:57.293 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:46:57.293 INFO:teuthology.orchestra.run.smithi110.stdout:Total 167 kB/s | 127 kB 00:00 2024-03-25T18:46:57.293 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:46:57.299 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:46:57.299 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:46:57.357 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:46:57.357 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:46:57.499 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:46:57.591 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:58.054 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:58.350 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:46:58.350 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:58.350 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:46:58.350 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:46:58.350 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:46:58.350 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:58.588 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2 2024-03-25T18:46:59.164 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:55 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:46:59.264 INFO:teuthology.orchestra.run.smithi110.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:59.319 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:46:59.320 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:46:59.320 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:46:59.395 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs-devel 2024-03-25T18:46:59.959 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:55 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:00.107 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:00.108 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:00.108 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T18:47:00.108 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 31 k 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 140 k 2024-03-25T18:47:00.109 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:47:00.547 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 71 kB/s | 31 kB 00:00 2024-03-25T18:47:00.548 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:00.548 INFO:teuthology.orchestra.run.smithi110.stdout:Total 71 kB/s | 31 kB 00:00 2024-03-25T18:47:00.548 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:47:00.552 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:47:00.552 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:47:00.579 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:47:00.580 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:47:00.701 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:47:00.796 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:01.160 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:01.428 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:01.428 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:01.429 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:47:01.429 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:01.429 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:01.429 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:01.590 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2 2024-03-25T18:47:02.151 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:58 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:02.247 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:47:02.298 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:02.299 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:47:02.299 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:02.348 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1 2024-03-25T18:47:02.911 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:58 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:03.007 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:47:03.059 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:03.060 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:47:03.060 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:03.112 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados 2024-03-25T18:47:03.238 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 350 kB/s | 3.1 MB 00:09 2024-03-25T18:47:03.239 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:03.239 INFO:teuthology.orchestra.run.smithi178.stdout:Total 350 kB/s | 3.1 MB 00:09 2024-03-25T18:47:03.240 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:47:03.250 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:47:03.251 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:47:03.328 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:47:03.328 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:47:03.665 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:03.761 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:47:03.778 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:47:03.811 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:03.812 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:47:03.812 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:03.847 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:03.865 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw 2024-03-25T18:47:03.872 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:03.872 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:47:03.872 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:47:03.872 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:47:03.872 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:47:03.872 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:04.421 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:00 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:04.517 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:47:04.569 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:04.570 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:47:04.570 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:04.622 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs 2024-03-25T18:47:05.028 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:05.028 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:05.028 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:47:05.028 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:05.028 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:05.029 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:47:05.206 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:05.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2024-03-25T18:47:05.306 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:47:05.358 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:05.359 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:47:05.359 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:05.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd 2024-03-25T18:47:05.831 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:47:05.979 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:47:05.979 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:47:05.979 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-03-25T18:47:05.979 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:47:05.979 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:47:05.979 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:47:05.979 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:05.980 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:47:05.980 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:47:05.980 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T18:47:05.980 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:05.980 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 171 k 2024-03-25T18:47:05.980 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 494 k 2024-03-25T18:47:05.980 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:47:05.999 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:06.095 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:47:06.147 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:06.148 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T18:47:06.148 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:06.204 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse 2024-03-25T18:47:06.708 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 236 kB/s | 171 kB 00:00 2024-03-25T18:47:06.708 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:06.708 INFO:teuthology.orchestra.run.smithi178.stdout:Total 235 kB/s | 171 kB 00:00 2024-03-25T18:47:06.709 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:47:06.717 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:47:06.717 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:47:06.762 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:06.775 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:47:06.776 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:47:06.917 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:06.917 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:06.917 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-03-25T18:47:06.917 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:06.917 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:47:06.917 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:47:06.917 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:06.917 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:47:06.918 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:06.918 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T18:47:06.918 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:06.918 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 85 k 2024-03-25T18:47:06.918 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 231 k 2024-03-25T18:47:06.918 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:47:06.922 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:47:07.057 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:07.516 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:07.776 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 99 kB/s | 85 kB 00:00 2024-03-25T18:47:07.776 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:07.776 INFO:teuthology.orchestra.run.smithi110.stdout:Total 99 kB/s | 85 kB 00:00 2024-03-25T18:47:07.777 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:47:07.784 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:47:07.784 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:47:07.835 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:07.835 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:07.836 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:47:07.836 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:07.836 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:07.836 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:47:07.841 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:47:07.841 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:47:07.982 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:47:08.083 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:08.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:47:08.486 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:08.655 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:47:08.785 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:08.785 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:08.785 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:47:08.785 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:08.785 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:08.785 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:08.807 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:47:08.808 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:47:08.808 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T18:47:08.808 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:47:08.808 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:47:08.808 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:47:08.808 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-25T18:47:08.808 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:47:08.809 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:08.809 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:47:08.809 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:47:08.809 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-03-25T18:47:08.809 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:47:08.809 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 91 M 2024-03-25T18:47:08.809 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 498 M 2024-03-25T18:47:08.809 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:47:09.027 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror 2024-03-25T18:47:09.594 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:05 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:09.748 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:09.748 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 3.1 M 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 13 M 2024-03-25T18:47:09.749 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:47:19.413 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 327 kB/s | 3.1 MB 00:09 2024-03-25T18:47:19.413 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:19.414 INFO:teuthology.orchestra.run.smithi110.stdout:Total 327 kB/s | 3.1 MB 00:09 2024-03-25T18:47:19.414 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:47:19.425 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:47:19.425 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:47:19.504 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:47:19.504 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:47:19.933 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:47:19.997 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:20.021 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:20.021 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:47:20.021 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:47:20.022 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:47:20.022 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:47:20.022 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:21.085 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:21.085 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:21.085 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:47:21.085 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:21.085 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:21.085 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:21.335 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd 2024-03-25T18:47:21.901 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:22.048 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:22.049 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:22.049 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-03-25T18:47:22.049 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 171 k 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 494 k 2024-03-25T18:47:22.050 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:47:22.816 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 224 kB/s | 171 kB 00:00 2024-03-25T18:47:22.817 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:22.817 INFO:teuthology.orchestra.run.smithi110.stdout:Total 223 kB/s | 171 kB 00:00 2024-03-25T18:47:22.817 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:47:22.825 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:47:22.826 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:47:22.883 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:47:22.884 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:47:23.103 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:47:23.214 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:23.633 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:23.884 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:47:23.885 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:23.885 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:47:23.885 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:23.885 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:23.885 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:47:24.101 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:47:24.674 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:47:24.828 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:47:24.829 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:24.829 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T18:47:24.829 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:24.829 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:47:24.829 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:47:24.829 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-03-25T18:47:24.830 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:47:24.830 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:24.830 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:47:24.830 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:47:24.830 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-03-25T18:47:24.830 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:47:24.831 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 91 M 2024-03-25T18:47:24.831 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 498 M 2024-03-25T18:47:24.831 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T18:48:09.260 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 380 kB/s | 22 MB 01:00 2024-03-25T18:48:25.964 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 376 kB/s | 22 MB 01:01 2024-03-25T18:49:53.118 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 425 kB/s | 68 MB 02:44 2024-03-25T18:49:53.119 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:49:53.119 INFO:teuthology.orchestra.run.smithi178.stdout:Total 565 kB/s | 91 MB 02:44 2024-03-25T18:49:53.119 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T18:49:53.123 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T18:49:53.123 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T18:49:53.717 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T18:49:53.717 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T18:49:56.834 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T18:50:03.941 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:50:04.010 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:50:07.095 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 430 kB/s | 68 MB 02:42 2024-03-25T18:50:07.096 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:50:07.097 INFO:teuthology.orchestra.run.smithi110.stdout:Total 572 kB/s | 91 MB 02:42 2024-03-25T18:50:07.097 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T18:50:07.101 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T18:50:07.101 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T18:50:07.319 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:50:07.319 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:50:07.581 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:50:07.582 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:50:07.582 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T18:50:07.582 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:07.582 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:07.582 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:50:07.582 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T18:50:07.693 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T18:50:07.693 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T18:50:07.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:50:08.394 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:08:02 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T18:50:08.548 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T18:50:08.548 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:50:08.548 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T18:50:08.548 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:50:08.548 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T18:50:08.549 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:50:08.549 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:50:08.549 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T18:50:08.549 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T18:50:08.549 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T18:50:08.549 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T18:50:08.549 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 416 M 2024-03-25T18:50:08.549 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.7 G 2024-03-25T18:50:08.550 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T18:50:10.862 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T18:50:18.027 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:50:18.180 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:50:21.593 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:50:21.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:50:21.868 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:50:21.868 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:50:21.868 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T18:50:21.868 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:21.868 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:21.868 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:50:21.868 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T18:50:22.089 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:50:22.655 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:08:18 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T18:50:22.803 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T18:50:22.804 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T18:50:22.805 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T18:50:22.805 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 416 M 2024-03-25T18:50:22.805 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.7 G 2024-03-25T18:50:22.805 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:01:41.018 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 615 kB/s | 416 MB 11:32 2024-03-25T19:01:41.019 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:01:41.019 INFO:teuthology.orchestra.run.smithi178.stdout:Total 615 kB/s | 416 MB 11:32 2024-03-25T19:01:41.019 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:01:41.022 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:01:41.022 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:01:43.497 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:01:43.497 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:02:25.488 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:02:25.584 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:02:30.635 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:02:30.921 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:02:30.921 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:02:30.921 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:02:30.921 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:02:30.921 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:02:30.921 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:02:31.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T19:02:31.787 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:20:25 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:02:31.944 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================== 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.4 M 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 6.4 M 2024-03-25T19:02:31.946 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:02:34.647 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 541 kB/s | 1.4 MB 00:02 2024-03-25T19:02:34.648 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:34.648 INFO:teuthology.orchestra.run.smithi178.stdout:Total 541 kB/s | 1.4 MB 00:02 2024-03-25T19:02:34.648 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:02:34.653 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:02:34.653 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:02:34.697 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:02:34.698 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:02:34.974 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:02:35.078 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:02:35.460 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:02:35.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:02:35.751 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:02:35.751 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:02:35.751 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:02:35.751 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:02:35.752 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:02:35.971 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T19:02:36.533 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:20:30 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:02:36.682 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:02:36.683 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 176 M 2024-03-25T19:02:36.684 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 753 M 2024-03-25T19:02:36.684 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:06:46.597 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 433 kB/s | 416 MB 16:23 2024-03-25T19:06:46.597 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:06:46.597 INFO:teuthology.orchestra.run.smithi110.stdout:Total 433 kB/s | 416 MB 16:23 2024-03-25T19:06:46.598 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:06:46.601 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:06:46.601 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:06:49.076 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:06:49.076 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:07:31.293 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:07:31.385 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:07:36.389 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:07:36.650 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:07:36.650 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:07:36.650 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:07:36.651 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:36.651 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:07:36.651 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:07:36.871 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T19:07:37.435 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:25:33 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:07:37.583 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:07:37.584 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:07:37.585 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.4 M 2024-03-25T19:07:37.585 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 6.4 M 2024-03-25T19:07:37.585 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:07:40.588 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 487 kB/s | 1.4 MB 00:03 2024-03-25T19:07:40.589 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:40.589 INFO:teuthology.orchestra.run.smithi110.stdout:Total 486 kB/s | 1.4 MB 00:03 2024-03-25T19:07:40.590 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:07:40.593 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:07:40.594 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:07:40.639 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:07:40.639 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:07:40.895 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:07:41.003 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:07:41.387 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:07:41.652 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:07:41.652 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:07:41.652 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:07:41.652 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:41.652 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:07:41.652 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:07:41.848 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T19:07:42.409 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:25:38 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:07:42.556 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:07:42.557 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:07:42.558 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 176 M 2024-03-25T19:07:42.558 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 753 M 2024-03-25T19:07:42.558 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:11:06.508 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 353 kB/s | 176 MB 08:29 2024-03-25T19:11:06.508 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:11:06.509 INFO:teuthology.orchestra.run.smithi178.stdout:Total 353 kB/s | 176 MB 08:29 2024-03-25T19:11:06.509 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:11:06.511 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:11:06.512 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:11:07.577 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:11:07.577 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:11:25.610 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:11:25.787 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:11:30.445 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:11:30.719 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:11:30.719 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:11:30.719 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:11:30.719 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:11:30.719 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:11:30.719 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:11:30.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:11:31.505 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:29:25 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:11:31.654 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 672 M 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 3.1 G 2024-03-25T19:11:31.655 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:13:54.952 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 483 kB/s | 176 MB 06:12 2024-03-25T19:13:54.952 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:54.952 INFO:teuthology.orchestra.run.smithi110.stdout:Total 483 kB/s | 176 MB 06:12 2024-03-25T19:13:54.953 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:13:54.955 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:13:54.956 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:13:56.016 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:13:56.016 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:14:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:14:14.223 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:14:18.892 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:14:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:14:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:14:19.148 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:14:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:14:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:14:19.148 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:14:19.339 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:14:19.905 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:32:15 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:14:20.052 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:14:20.052 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:14:20.052 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:14:20.052 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:14:20.052 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:14:20.052 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:14:20.052 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:14:20.053 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:14:20.053 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:14:20.053 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:14:20.053 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:14:20.053 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 672 M 2024-03-25T19:14:20.053 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 3.1 G 2024-03-25T19:14:20.054 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:27:24.727 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 877 kB/s | 672 MB 13:04 2024-03-25T19:27:24.727 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:24.727 INFO:teuthology.orchestra.run.smithi110.stdout:Total 877 kB/s | 672 MB 13:04 2024-03-25T19:27:24.728 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:27:24.731 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:27:24.731 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:27:28.697 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:27:28.697 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:28:31.813 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 675 kB/s | 672 MB 17:00 2024-03-25T19:28:31.813 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:31.813 INFO:teuthology.orchestra.run.smithi178.stdout:Total 675 kB/s | 672 MB 17:00 2024-03-25T19:28:31.813 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:28:31.817 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:28:31.817 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:28:35.780 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:28:35.780 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:28:39.739 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:28:39.896 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:28:47.721 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:28:47.971 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:28:47.971 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:28:47.971 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:28:47.972 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:47.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:28:47.972 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:28:48.276 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:28:48.841 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:46:44 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:28:48.939 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:28:48.990 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:28:48.991 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-03-25T19:28:48.991 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:28:49.064 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:28:49.624 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:46:45 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:28:49.772 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:28:49.772 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:28:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:28:49.772 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:28:49.772 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:28:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:28:49.772 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:28:49.773 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:28:49.773 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:28:49.773 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:28:49.773 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:28:49.773 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 20 M 2024-03-25T19:28:49.773 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 103 M 2024-03-25T19:28:49.773 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:29:08.032 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 20 MB 00:18 2024-03-25T19:29:08.032 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:08.033 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.1 MB/s | 20 MB 00:18 2024-03-25T19:29:08.033 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:29:08.037 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:29:08.037 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:29:08.187 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:29:08.188 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:29:10.421 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:29:10.539 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:11.517 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:11.779 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:11.779 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:29:11.780 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:29:11.780 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:11.780 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:29:11.780 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:29:11.961 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:29:12.522 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:47:08 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:29:12.669 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:29:12.670 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:29:12.670 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:29:12.670 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:29:12.670 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:29:12.670 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:29:12.670 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:29:12.671 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:29:12.671 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:29:12.671 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:29:12.671 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:29:12.671 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 48 M 2024-03-25T19:29:12.671 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 173 M 2024-03-25T19:29:12.671 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:29:46.502 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:29:46.658 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:49.755 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 48 MB 00:37 2024-03-25T19:29:49.756 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:49.757 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.3 MB/s | 48 MB 00:37 2024-03-25T19:29:49.757 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:29:49.760 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:29:49.760 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:29:50.071 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:29:50.071 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:29:54.778 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:29:55.004 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:56.373 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:56.572 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:56.625 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:56.625 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:29:56.625 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:29:56.625 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:56.625 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:29:56.625 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:29:56.801 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:29:56.836 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:56.836 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:29:56.836 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:29:56.836 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:56.836 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:29:56.836 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:29:57.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:29:57.365 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:47:53 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:29:57.514 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:29:57.515 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:29:57.516 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:29:57.516 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 99 M 2024-03-25T19:29:57.516 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 449 M 2024-03-25T19:29:57.516 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:29:57.696 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:47:51 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:29:57.793 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:29:57.845 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:29:57.846 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-03-25T19:29:57.846 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:29:57.901 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:29:58.462 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:47:52 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:29:58.612 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:29:58.613 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:29:58.613 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:29:58.613 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:29:58.613 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:29:58.613 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:29:58.613 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:29:58.613 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:29:58.614 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:29:58.614 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:29:58.614 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:29:58.614 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 20 M 2024-03-25T19:29:58.614 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 103 M 2024-03-25T19:29:58.614 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:30:05.769 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 2.8 MB/s | 20 MB 00:07 2024-03-25T19:30:05.770 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:05.770 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.8 MB/s | 20 MB 00:07 2024-03-25T19:30:05.771 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:30:05.775 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:30:05.775 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:30:05.925 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:30:05.925 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:30:08.152 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:30:08.268 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:09.210 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:30:09.476 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:30:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:30:09.476 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:30:09.476 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:30:09.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:30:10.216 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:48:04 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:30:10.363 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:30:10.364 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:30:10.364 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 48 M 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 173 M 2024-03-25T19:30:10.365 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:30:42.575 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.5 MB/s | 48 MB 00:32 2024-03-25T19:30:42.576 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:42.576 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.5 MB/s | 48 MB 00:32 2024-03-25T19:30:42.576 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:30:42.580 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:30:42.580 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:30:42.895 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:30:42.895 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:30:47.639 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:30:47.722 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:49.082 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:49.343 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:49.343 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:30:49.343 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:30:49.343 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:30:49.343 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:30:49.343 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:30:49.538 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:30:50.102 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:48:44 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:30:50.251 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:30:50.251 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:30:50.251 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 99 M 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 449 M 2024-03-25T19:30:50.252 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:32:01.663 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 813 kB/s | 99 MB 02:04 2024-03-25T19:32:01.663 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:01.663 INFO:teuthology.orchestra.run.smithi110.stdout:Total 813 kB/s | 99 MB 02:04 2024-03-25T19:32:01.664 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:32:01.667 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:32:01.667 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:32:02.276 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:32:02.277 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:32:12.533 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:32:12.652 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:15.587 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:15.857 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:15.857 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:32:15.858 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:32:15.858 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:15.858 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:32:15.858 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:32:16.076 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:32:16.633 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:50:12 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:32:16.781 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:32:16.782 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:32:16.783 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 287 M 2024-03-25T19:32:16.783 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.2 G 2024-03-25T19:32:16.783 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:32:29.345 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 99 MB 01:39 2024-03-25T19:32:29.346 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:29.346 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.0 MB/s | 99 MB 01:39 2024-03-25T19:32:29.346 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:32:29.349 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:32:29.349 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:32:29.960 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:32:29.960 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:32:40.250 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:32:40.342 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:43.353 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:43.605 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:43.606 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:32:43.606 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:32:43.606 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:43.606 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:32:43.606 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:32:43.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:32:44.353 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:50:38 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:32:44.503 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:32:44.503 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:32:44.503 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:32:44.503 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 287 M 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.2 G 2024-03-25T19:32:44.504 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:35:48.900 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.6 MB/s | 287 MB 03:04 2024-03-25T19:35:48.900 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:48.901 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.6 MB/s | 287 MB 03:04 2024-03-25T19:35:48.901 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:35:48.904 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:35:48.904 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:35:50.609 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:35:50.609 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:36:20.171 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:36:20.434 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:28.020 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:28.304 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:28.304 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:28.305 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:36:28.305 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:28.305 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:28.305 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:36:28.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:36:29.082 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:54:23 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:29.231 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:36:29.232 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:29.232 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 5.9 M 2024-03-25T19:36:29.232 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 28 M 2024-03-25T19:36:29.232 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:36:32.921 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 287 MB 04:16 2024-03-25T19:36:32.921 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:32.922 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.1 MB/s | 287 MB 04:16 2024-03-25T19:36:32.922 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:36:32.925 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:36:32.925 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:36:34.638 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:36:34.638 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:36:36.195 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 874 kB/s | 5.9 MB 00:06 2024-03-25T19:36:36.196 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:36.197 INFO:teuthology.orchestra.run.smithi178.stdout:Total 874 kB/s | 5.9 MB 00:06 2024-03-25T19:36:36.197 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:36:36.201 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:36:36.201 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:36:36.273 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:36:36.274 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:36:37.001 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:36:37.102 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:37.705 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:37.979 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:37.980 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:37.980 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:36:37.980 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:37.980 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:37.980 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:36:38.167 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:36:38.731 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:54:32 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:36:38.883 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 109 k 2024-03-25T19:36:38.884 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 440 k 2024-03-25T19:36:38.885 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:36:39.415 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 205 kB/s | 109 kB 00:00 2024-03-25T19:36:39.415 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:39.416 INFO:teuthology.orchestra.run.smithi178.stdout:Total 204 kB/s | 109 kB 00:00 2024-03-25T19:36:39.416 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:36:39.419 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:36:39.419 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:36:39.455 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:36:39.455 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:36:39.570 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:36:39.663 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:36:40.049 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:36:40.328 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:36:40.328 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:40.328 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:36:40.328 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:40.328 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:40.328 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:36:40.507 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:36:41.067 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:54:35 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:36:41.214 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:36:41.215 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:41.216 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 5.4 M 2024-03-25T19:36:41.216 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 25 M 2024-03-25T19:36:41.216 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:36:47.942 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 815 kB/s | 5.4 MB 00:06 2024-03-25T19:36:47.943 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:47.943 INFO:teuthology.orchestra.run.smithi178.stdout:Total 815 kB/s | 5.4 MB 00:06 2024-03-25T19:36:47.943 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:36:47.947 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:36:47.947 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:36:48.014 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:36:48.015 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:36:48.672 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:36:48.790 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:49.316 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:49.586 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:49.586 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:49.586 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:36:49.586 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:49.586 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:49.586 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:36:49.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2-debuginfo 2024-03-25T19:36:50.337 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:54:44 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:36:50.488 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:36:50.488 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:50.488 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:36:50.488 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 44 M 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 253 M 2024-03-25T19:36:50.489 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:37:03.992 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:37:04.511 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:09.158 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:09.437 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:09.437 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:09.438 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:37:09.438 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:09.438 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:09.438 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:37:09.647 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:37:10.207 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:55:06 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:37:10.355 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:37:10.356 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:10.357 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 5.9 M 2024-03-25T19:37:10.357 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 28 M 2024-03-25T19:37:10.357 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:37:13.614 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.8 MB/s | 5.9 MB 00:03 2024-03-25T19:37:13.614 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:13.615 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.8 MB/s | 5.9 MB 00:03 2024-03-25T19:37:13.615 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:37:13.619 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:37:13.619 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:37:13.691 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:37:13.691 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:37:14.425 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:37:14.528 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:15.055 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:15.310 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:15.310 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:15.310 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:37:15.310 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:15.311 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:15.311 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:37:15.503 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:37:16.061 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:55:12 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:37:16.212 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:37:16.212 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:37:16.212 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:16.212 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:37:16.212 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:37:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:16.213 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:37:16.213 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:16.213 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:37:16.213 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:16.213 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 109 k 2024-03-25T19:37:16.213 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 440 k 2024-03-25T19:37:16.213 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:37:16.909 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 156 kB/s | 109 kB 00:00 2024-03-25T19:37:16.909 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:16.909 INFO:teuthology.orchestra.run.smithi110.stdout:Total 156 kB/s | 109 kB 00:00 2024-03-25T19:37:16.910 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:37:16.913 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:37:16.913 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:37:16.949 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:37:16.949 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:37:17.061 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:37:17.149 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:17.524 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:17.777 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:17.777 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:17.777 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:37:17.777 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:17.777 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:17.777 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:37:17.964 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:37:18.521 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:55:14 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:37:18.669 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:37:18.670 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:18.671 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 5.4 M 2024-03-25T19:37:18.671 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 25 M 2024-03-25T19:37:18.671 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:37:25.223 INFO:teuthology.orchestra.run.smithi178.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.3 MB/s | 44 MB 00:34 2024-03-25T19:37:25.223 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:25.224 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.3 MB/s | 44 MB 00:34 2024-03-25T19:37:25.224 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:37:25.228 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:37:25.228 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:37:25.528 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:37:25.528 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:37:30.629 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:37:30.719 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 455 kB/s | 5.4 MB 00:12 2024-03-25T19:37:30.720 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:30.720 INFO:teuthology.orchestra.run.smithi110.stdout:Total 455 kB/s | 5.4 MB 00:12 2024-03-25T19:37:30.721 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:37:30.725 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:37:30.725 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:37:30.730 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:30.792 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:37:30.793 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:37:31.456 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:37:31.560 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:32.130 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:37:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:32.393 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:37:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:32.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:32.393 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:37:32.578 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2-debuginfo 2024-03-25T19:37:32.605 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:32.895 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:32.895 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:37:32.895 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:37:32.895 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:32.896 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:37:32.896 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:37:33.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:37:33.135 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:55:29 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:37:33.283 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:37:33.283 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:33.283 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:37:33.283 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 44 M 2024-03-25T19:37:33.284 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 253 M 2024-03-25T19:37:33.285 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:37:33.664 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:55:27 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:37:33.814 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:37:33.815 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:37:33.816 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 52 M 2024-03-25T19:37:33.816 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 308 M 2024-03-25T19:37:33.816 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:37:59.162 INFO:teuthology.orchestra.run.smithi110.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.7 MB/s | 44 MB 00:25 2024-03-25T19:37:59.162 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:59.162 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.7 MB/s | 44 MB 00:25 2024-03-25T19:37:59.163 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:37:59.167 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:37:59.167 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:37:59.466 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:37:59.467 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:38:04.528 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:38:04.626 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:05.728 INFO:teuthology.orchestra.run.smithi178.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.6 MB/s | 52 MB 00:31 2024-03-25T19:38:05.729 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:05.730 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.6 MB/s | 52 MB 00:31 2024-03-25T19:38:05.730 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:38:05.734 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:38:05.734 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:38:06.079 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:38:06.080 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:38:06.422 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:06.686 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:06.687 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:06.687 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:38:06.687 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:06.687 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:06.687 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:38:06.875 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:38:07.437 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:56:03 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:38:07.584 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:07.585 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:38:07.586 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:07.586 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 52 M 2024-03-25T19:38:07.586 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 308 M 2024-03-25T19:38:07.586 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:38:12.063 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:38:12.169 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:14.384 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:14.650 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:14.650 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:14.650 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:38:14.650 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:14.650 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:14.650 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:38:14.848 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:38:15.408 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:56:09 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:38:15.558 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:38:15.558 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:15.558 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:38:15.558 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:15.558 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 615 k 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.1 M 2024-03-25T19:38:15.559 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:38:16.472 INFO:teuthology.orchestra.run.smithi178.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 674 kB/s | 615 kB 00:00 2024-03-25T19:38:16.473 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:16.473 INFO:teuthology.orchestra.run.smithi178.stdout:Total 673 kB/s | 615 kB 00:00 2024-03-25T19:38:16.473 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:38:16.477 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:38:16.477 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:38:16.515 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:38:16.516 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:38:16.688 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:38:16.796 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:17.215 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:17.487 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:17.487 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:17.487 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:38:17.487 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:17.487 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:17.487 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:38:17.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:38:18.226 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:56:12 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:38:18.378 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:38:18.378 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:18.378 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 771 k 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.7 M 2024-03-25T19:38:18.379 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:38:19.950 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 491 kB/s | 771 kB 00:01 2024-03-25T19:38:19.950 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:19.951 INFO:teuthology.orchestra.run.smithi178.stdout:Total 490 kB/s | 771 kB 00:01 2024-03-25T19:38:19.951 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:38:19.955 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:38:19.955 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:38:19.997 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:38:19.997 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:38:20.178 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:38:20.274 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:38:20.640 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:38:20.910 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:38:20.911 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:20.911 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:38:20.911 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:20.911 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:20.911 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:38:21.122 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:38:21.685 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:56:15 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:38:21.834 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:38:21.835 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:21.835 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:38:21.835 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:21.835 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:38:21.835 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:38:21.835 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:21.835 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:38:21.835 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:21.836 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:38:21.836 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:21.836 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.1 M 2024-03-25T19:38:21.836 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 4.3 M 2024-03-25T19:38:21.836 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:38:22.970 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 1.0 MB/s | 1.1 MB 00:01 2024-03-25T19:38:22.970 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:22.971 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.0 MB/s | 1.1 MB 00:01 2024-03-25T19:38:22.971 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:38:22.974 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:38:22.975 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:38:23.016 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:38:23.016 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:38:23.231 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:38:23.327 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:23.736 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:24.009 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:24.009 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:24.009 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:38:24.009 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:24.009 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:24.010 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:38:24.237 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:38:24.795 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:56:18 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:38:24.944 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:38:24.944 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:24.944 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:38:24.944 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:24.944 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:38:24.944 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:38:24.945 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:24.945 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:38:24.945 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:24.945 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:38:24.945 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:24.945 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 337 k 2024-03-25T19:38:24.945 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.0 M 2024-03-25T19:38:24.945 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:38:25.823 INFO:teuthology.orchestra.run.smithi178.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 385 kB/s | 337 kB 00:00 2024-03-25T19:38:25.823 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:25.824 INFO:teuthology.orchestra.run.smithi178.stdout:Total 384 kB/s | 337 kB 00:00 2024-03-25T19:38:25.824 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:38:25.827 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:38:25.828 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:38:25.865 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:38:25.866 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:38:26.001 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:38:26.097 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:26.481 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:26.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:26.750 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:26.750 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:38:26.750 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:26.750 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:26.750 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:38:26.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:38:27.522 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:56:21 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:38:27.672 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 509 k 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.5 M 2024-03-25T19:38:27.673 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:38:29.017 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 379 kB/s | 509 kB 00:01 2024-03-25T19:38:29.017 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:29.018 INFO:teuthology.orchestra.run.smithi178.stdout:Total 379 kB/s | 509 kB 00:01 2024-03-25T19:38:29.018 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:38:29.022 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:38:29.022 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:38:29.062 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:38:29.063 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:38:29.225 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:38:29.349 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:29.750 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:30.026 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:30.026 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:30.026 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:38:30.026 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:30.026 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:30.026 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:38:30.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:38:30.780 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:56:24 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:38:30.929 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:38:30.930 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:38:30.931 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:38:30.931 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 52 M 2024-03-25T19:38:30.931 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 298 M 2024-03-25T19:38:30.931 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:38:32.704 INFO:teuthology.orchestra.run.smithi110.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.1 MB/s | 52 MB 00:25 2024-03-25T19:38:32.704 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:32.705 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.1 MB/s | 52 MB 00:25 2024-03-25T19:38:32.705 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:38:32.709 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:38:32.709 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:38:33.054 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:38:33.054 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:38:39.019 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:38:39.119 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:41.258 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:41.524 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:41.524 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:41.524 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:38:41.524 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:41.524 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:41.524 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:38:41.716 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:38:42.273 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:56:38 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:38:42.422 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 615 k 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.1 M 2024-03-25T19:38:42.423 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:38:43.507 INFO:teuthology.orchestra.run.smithi110.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 568 kB/s | 615 kB 00:01 2024-03-25T19:38:43.508 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:43.508 INFO:teuthology.orchestra.run.smithi110.stdout:Total 567 kB/s | 615 kB 00:01 2024-03-25T19:38:43.508 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:38:43.511 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:38:43.512 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:38:43.550 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:38:43.550 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:38:43.716 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:38:43.822 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:44.206 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:44.467 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:44.467 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:44.468 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:38:44.468 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:44.468 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:44.468 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:38:44.659 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:38:45.222 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:56:41 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:38:45.373 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:38:45.373 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:38:45.374 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:45.375 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 771 k 2024-03-25T19:38:45.375 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.7 M 2024-03-25T19:38:45.375 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:38:46.235 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 896 kB/s | 771 kB 00:00 2024-03-25T19:38:46.236 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:46.236 INFO:teuthology.orchestra.run.smithi110.stdout:Total 895 kB/s | 771 kB 00:00 2024-03-25T19:38:46.236 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:38:46.240 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:38:46.240 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:38:46.280 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:38:46.280 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:38:46.458 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:38:46.568 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:38:46.963 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:38:47.242 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:38:47.242 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:47.243 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:38:47.243 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:47.243 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:47.243 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:38:47.448 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:38:48.010 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:56:44 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:38:48.160 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:38:48.160 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:48.160 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:38:48.160 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.1 M 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 4.3 M 2024-03-25T19:38:48.161 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:38:49.274 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 1.0 MB/s | 1.1 MB 00:01 2024-03-25T19:38:49.275 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:49.275 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.0 MB/s | 1.1 MB 00:01 2024-03-25T19:38:49.275 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:38:49.279 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:38:49.279 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:38:49.320 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:38:49.320 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:38:49.536 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:38:49.631 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:50.031 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:50.298 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:50.299 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:50.299 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:38:50.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:50.299 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:50.299 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:38:50.500 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:38:51.065 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:56:47 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:38:51.213 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:38:51.214 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:51.214 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 337 k 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.0 M 2024-03-25T19:38:51.215 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:38:52.094 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 384 kB/s | 337 kB 00:00 2024-03-25T19:38:52.095 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:52.095 INFO:teuthology.orchestra.run.smithi110.stdout:Total 383 kB/s | 337 kB 00:00 2024-03-25T19:38:52.095 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:38:52.099 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:38:52.099 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:38:52.139 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:38:52.139 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:38:52.285 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:38:52.385 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:52.762 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:53.024 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:38:53.024 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:53.024 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:38:53.024 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:53.024 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:53.024 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:38:53.228 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:38:53.791 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:56:49 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:38:53.941 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:53.942 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 509 k 2024-03-25T19:38:53.942 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.5 M 2024-03-25T19:38:53.942 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:38:54.655 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 714 kB/s | 509 kB 00:00 2024-03-25T19:38:54.656 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:54.656 INFO:teuthology.orchestra.run.smithi110.stdout:Total 712 kB/s | 509 kB 00:00 2024-03-25T19:38:54.656 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:38:54.660 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:38:54.660 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:38:54.697 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:38:54.698 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:38:54.862 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:38:54.980 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:55.330 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:55.598 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:55.599 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:55.599 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:38:55.599 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:55.599 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:55.599 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:38:55.790 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:38:56.353 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:56:52 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:38:56.503 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:38:56.504 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:38:56.505 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 52 M 2024-03-25T19:38:56.505 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 298 M 2024-03-25T19:38:56.505 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:39:05.990 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 52 MB 00:35 2024-03-25T19:39:05.991 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:05.991 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.5 MB/s | 52 MB 00:35 2024-03-25T19:39:05.991 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:39:05.995 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:39:05.995 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:39:06.330 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:39:06.331 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:39:08.058 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 4.5 MB/s | 52 MB 00:11 2024-03-25T19:39:08.059 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:08.059 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.5 MB/s | 52 MB 00:11 2024-03-25T19:39:08.059 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:39:08.063 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:39:08.063 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:39:08.397 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:39:08.398 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:39:12.264 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:39:12.529 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:14.276 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:39:14.639 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:14.647 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:14.924 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:14.925 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:14.925 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:39:14.925 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:14.925 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:14.925 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:39:15.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:39:15.803 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:57:09 ago on Mon 25 Mar 2024 06:42:06 PM UTC. 2024-03-25T19:39:15.952 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.3 M 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 5.5 M 2024-03-25T19:39:15.953 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-25T19:39:16.725 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.6 MB/s | 1.3 MB 00:00 2024-03-25T19:39:16.725 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:16.725 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.6 MB/s | 1.3 MB 00:00 2024-03-25T19:39:16.726 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-25T19:39:16.729 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-25T19:39:16.729 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-25T19:39:16.737 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:16.771 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-25T19:39:16.772 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-25T19:39:17.007 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-25T19:39:17.008 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:17.008 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:17.008 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:39:17.008 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:17.008 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:17.008 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:39:17.098 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:17.200 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:39:17.527 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:17.757 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:57:13 ago on Mon 25 Mar 2024 06:42:04 PM UTC. 2024-03-25T19:39:17.799 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:17.799 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:17.800 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-25T19:39:17.800 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:17.800 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:17.800 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-25T19:39:17.907 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-03-25T19:39:17.907 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:39:17.907 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-03-25T19:39:17.907 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:39:17.908 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-03-25T19:39:17.908 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:39:17.908 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:17.908 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-03-25T19:39:17.908 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-03-25T19:39:17.908 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-03-25T19:39:17.908 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:17.908 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.3 M 2024-03-25T19:39:17.909 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 5.5 M 2024-03-25T19:39:17.909 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-03-25T19:39:18.012 DEBUG:teuthology.parallel:result is None 2024-03-25T19:39:19.160 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 1.3 MB 00:01 2024-03-25T19:39:19.160 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:19.161 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.0 MB/s | 1.3 MB 00:01 2024-03-25T19:39:19.161 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-03-25T19:39:19.165 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-03-25T19:39:19.165 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-03-25T19:39:19.211 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-03-25T19:39:19.212 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-03-25T19:39:19.458 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-03-25T19:39:19.576 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:19.986 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:20.253 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:20.253 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:20.253 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-03-25T19:39:20.253 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:20.253 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:20.253 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-03-25T19:39:20.446 DEBUG:teuthology.parallel:result is None 2024-03-25T19:39:20.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:39:20.668 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:39:20.688 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:20.689 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:20.689 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:39:20.691 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:39:20.838 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:39:20.863 INFO:teuthology.orchestra.run.smithi178.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:20.863 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:20.863 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:39:20.864 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:39:20.865 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:20.865 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:39:20.892 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:20.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:39:20.933 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:39:20.933 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:20.933 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:39:20.960 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:39:21.027 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:21.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:39:21.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:39:21.125 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:39:21.126 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:21.126 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:39:21.152 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:39:21.219 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:21.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:39:21.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:39:21.316 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:39:21.316 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:21.316 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:39:21.343 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:39:21.410 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:21.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:39:21.437 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:39:21.506 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:39:21.589 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:39:21.589 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2024-03-25T19:39:21.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /var/log/ceph 2024-03-25T19:39:21.619 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:39:21.619 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:39:21.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:39:21.686 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:39:21.686 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:39:21.725 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:39:21.764 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:39:21.764 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:39:21.765 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621333', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQnb3SNdRoZlsx73/yAyHFAb4x75q5beKbDacY86ZZwjKwAPqjOPSE/sCU7yAAbXbRFZY8kYDjjaHh33oivqv4=', 'smithi178.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM1L1qYqDlKANLXcEsGewMeo/VqOsIiqfBeFmUTo5wGc3fWIIJIbguckv6PzY9sD8SbC2cqXjydLyGeYjcWgBQQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:39:21.765 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:39:21.792 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:39:21.809 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:39:21.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:39:21.876 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:21.876 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:39:21.925 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:39:21.925 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-25 18:47:20.271770741 +0000 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-25 18:40:35.212574485 +0000 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-25 18:40:35.212574485 +0000 2024-03-25T19:39:21.980 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-25 18:40:35.212574485 +0000 2024-03-25T19:39:21.981 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:39:22.045 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-25T19:39:22.046 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-25T19:39:22.046 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.4908e-05 s, 6.8 MB/s 2024-03-25T19:39:22.047 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:39:22.102 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-25 18:47:20.271770741 +0000 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-25 18:40:35.533569385 +0000 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-25 18:40:35.533569385 +0000 2024-03-25T19:39:22.157 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-25 18:40:35.533569385 +0000 2024-03-25T19:39:22.157 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:39:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-25T19:39:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-25T19:39:22.222 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000103353 s, 5.0 MB/s 2024-03-25T19:39:22.223 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:39:22.278 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-03-25T19:39:22.333 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:39:22.334 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:22.334 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:39:22.334 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:22.334 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:22.334 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-25 18:47:20.272770725 +0000 2024-03-25T19:39:22.334 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-25 18:40:35.864564126 +0000 2024-03-25T19:39:22.334 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-25 18:40:35.864564126 +0000 2024-03-25T19:39:22.334 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-25 18:40:35.864564126 +0000 2024-03-25T19:39:22.334 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:39:22.399 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-25T19:39:22.400 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-25T19:39:22.400 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000101063 s, 5.1 MB/s 2024-03-25T19:39:22.401 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:39:22.456 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-03-25T19:39:22.510 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:39:22.511 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:22.511 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:39:22.511 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:22.511 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:22.511 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-03-25 18:47:20.272770725 +0000 2024-03-25T19:39:22.511 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-03-25 18:40:36.201558771 +0000 2024-03-25T19:39:22.511 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-03-25 18:40:36.202558755 +0000 2024-03-25T19:39:22.511 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-03-25 18:40:36.201558771 +0000 2024-03-25T19:39:22.511 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:39:22.575 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-03-25T19:39:22.575 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-03-25T19:39:22.575 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.7411e-05 s, 6.6 MB/s 2024-03-25T19:39:22.576 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:39:22.631 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:22.632 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:22.632 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:39:22.649 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:39:22.649 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-03-25T19:39:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:39:22.705 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:39:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-25 18:47:04.130703097 +0000 2024-03-25T19:39:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-25 18:41:11.293243751 +0000 2024-03-25T19:39:22.706 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-25 18:41:11.293243751 +0000 2024-03-25T19:39:22.706 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-25 18:41:11.293243751 +0000 2024-03-25T19:39:22.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:39:22.772 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-25T19:39:22.772 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-25T19:39:22.772 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000108953 s, 4.7 MB/s 2024-03-25T19:39:22.774 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:39:22.831 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-03-25T19:39:22.888 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:39:22.888 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:22.888 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:39:22.888 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:22.889 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:22.889 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-25 18:47:04.131703079 +0000 2024-03-25T19:39:22.889 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-25 18:41:11.620237681 +0000 2024-03-25T19:39:22.889 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-25 18:41:11.620237681 +0000 2024-03-25T19:39:22.889 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-25 18:41:11.620237681 +0000 2024-03-25T19:39:22.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:39:22.954 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-25T19:39:22.954 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-25T19:39:22.955 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000102941 s, 5.0 MB/s 2024-03-25T19:39:22.956 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:39:23.012 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-25 18:47:04.131703079 +0000 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-25 18:41:11.945231648 +0000 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-25 18:41:11.945231648 +0000 2024-03-25T19:39:23.067 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-25 18:41:11.945231648 +0000 2024-03-25T19:39:23.068 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:39:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-25T19:39:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-25T19:39:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000113837 s, 4.5 MB/s 2024-03-25T19:39:23.134 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:39:23.189 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-25 18:47:04.132703060 +0000 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-25 18:41:12.292225206 +0000 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-25 18:41:12.292225206 +0000 2024-03-25T19:39:23.244 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-25 18:41:12.292225206 +0000 2024-03-25T19:39:23.244 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:39:23.309 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-25T19:39:23.309 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-25T19:39:23.309 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 8.0405e-05 s, 6.4 MB/s 2024-03-25T19:39:23.310 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:39:23.365 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:23.366 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:39:23.366 INFO:tasks.ceph:Generating config... 2024-03-25T19:39:23.367 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:39:23.367 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:39:23.367 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:39:23.367 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:39:23.367 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:39:23.367 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:39:23.367 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:39:23.367 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:39:23.368 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:39:23.368 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:39:23.368 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:39:23.369 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:39:23.369 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:39:23.369 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:39:23.369 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:39:23.369 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:39:23.370 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:39:23.370 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:39:23.370 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:23.370 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:39:23.370 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:39:23.370 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:39:23.370 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:39:23.371 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:39:23.372 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:39:23.372 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:39:23.372 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:39:23.372 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:39:23.445 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:39:23.448 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:39:23.495 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:23.522 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.110:6789]'), ('mon.c', '[v1:172.21.15.110:6790]'), ('mon.b', '[v1:172.21.15.178:6789]')] 2024-03-25T19:39:23.523 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.110:6789],[v1:172.21.15.110:6790],[v1:172.21.15.178:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:39:23.523 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:23.524 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:39:23.577 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.110:6789]' --addv c '[v1:172.21.15.110:6790]' --addv b '[v1:172.21.15.178:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = quincy 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 17 (quincy) 2024-03-25T19:39:23.662 INFO:teuthology.orchestra.run.smithi110.stdout:election_strategy: 3 2024-03-25T19:39:23.663 INFO:teuthology.orchestra.run.smithi110.stdout:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:39:23.663 INFO:teuthology.orchestra.run.smithi110.stdout:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:39:23.663 INFO:teuthology.orchestra.run.smithi110.stdout:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:39:23.663 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:39:23.664 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:39:23.678 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 042bf057-0adf-4f81-8758-4792528254d3... 2024-03-25T19:39:23.679 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:39:23.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2024-03-25T19:39:23.768 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2024-03-25T19:39:23.769 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.770 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "[v1:172.21.15.110:6789],[v1:172.21.15.110:6790],[v1:172.21.15.178:6789]" 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 3 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: ms bind msgr2 = False 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 5000 2024-03-25T19:39:23.771 INFO:teuthology.orchestra.run.smithi110.stdout: ms type = async 2024-03-25T19:39:23.772 INFO:teuthology.orchestra.run.smithi110.stdout: osd heartbeat grace = 80 2024-03-25T19:39:23.772 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:39:23.772 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.772 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2024-03-25T19:39:23.772 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2024-03-25T19:39:23.772 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.772 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2024-03-25T19:39:23.772 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug trim objects = true 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2024-03-25T19:39:23.773 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2024-03-25T19:39:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: bdev async discard = True 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: bdev enable discard = True 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore allocator = stupid 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 1 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 1 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:23.775 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd class default list = * 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd class load list = * 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd client watch timeout = 120 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd fast shutdown = False 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd max markdown count = 1000 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2024-03-25T19:39:23.776 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: osd shutdown pgref assert = True 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:23.777 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2024-03-25T19:39:23.778 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:39:23.779 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon min osdmap epochs = 50 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd crush smoke test = False 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune interval = 2 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune min = 15 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2024-03-25T19:39:23.780 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on pool no app = False 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: paxos service trim min = 10 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: debug objecter = 20 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout: debug rados = 20 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2024-03-25T19:39:23.781 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2024-03-25T19:39:23.782 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-25T19:39:23.782 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-25T19:39:23.782 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:39:23.782 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-25T19:39:23.782 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.782 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-25T19:39:23.782 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.782 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-25T19:39:23.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-25T19:39:23.784 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "[v1:172.21.15.110:6789],[v1:172.21.15.110:6790],[v1:172.21.15.178:6789]" 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: mon client directed command retry = 5 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: mon election default strategy = 3 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: ms bind msgr2 = False 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: ms inject socket failures = 5000 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: ms type = async 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: osd heartbeat grace = 80 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:39:23.785 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-25T19:39:23.786 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-25T19:39:23.787 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = stupid 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore zero block detection = True 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: debug monc = 20 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 10 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:39:23.788 INFO:teuthology.orchestra.run.smithi178.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd class default list = * 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd class load list = * 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd client watch timeout = 120 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify cached snaps = True 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify missing on start = True 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd fast shutdown = False 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd max markdown count = 1000 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock override recovery settings = True 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd op queue = debug_random 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd op queue cut off = debug_random 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: osd shutdown pgref assert = True 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-25T19:39:23.789 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-25T19:39:23.790 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-25T19:39:23.791 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: mon min osdmap epochs = 50 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd crush smoke test = False 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune interval = 2 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune min = 15 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: mon scrub interval = 300 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on pool no app = False 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: paxos service trim min = 10 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:39:23.792 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:39:23.793 INFO:teuthology.orchestra.run.smithi178.stdout: debug monc = 20 2024-03-25T19:39:23.793 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-25T19:39:23.793 INFO:teuthology.orchestra.run.smithi178.stdout: debug objecter = 20 2024-03-25T19:39:23.793 INFO:teuthology.orchestra.run.smithi178.stdout: debug rados = 20 2024-03-25T19:39:23.793 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-25T19:39:23.793 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-25T19:39:23.793 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-25T19:39:23.794 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:39:23.794 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:39:23.850 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:39:23.850 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:23.850 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:39:23.865 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:23.865 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:39:23.920 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:39:23.920 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:23.920 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:39:23.920 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:23.999 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:23.999 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:24.054 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:39:24.054 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:24.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:39:24.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:24.094 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:24.094 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:24.149 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:39:24.149 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:39:24.149 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:39:24.203 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:39:24.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:39:24.296 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:39:24.298 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:39:24.299 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:39:24.301 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:39:24.343 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:39:24.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:39:24.400 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:39:24.416 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:39:24.416 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:39:24.416 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:24.443 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:24.443 INFO:tasks.ceph:role: osd.0 2024-03-25T19:39:24.443 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:39:24.443 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:39:24.509 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:24.509 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:24.510 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:24.510 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:24.510 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:24.510 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:24.510 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:24.510 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:24.510 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:24.510 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:24.542 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-03-25T19:39:24.543 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-03-25T19:39:24.544 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:24.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:24.686 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:24.753 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:24.753 INFO:tasks.ceph:role: osd.1 2024-03-25T19:39:24.753 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:39:24.753 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:24.819 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:24.820 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:24.851 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-03-25T19:39:24.853 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-03-25T19:39:24.853 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:24.924 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:24.995 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:25.061 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:25.062 INFO:tasks.ceph:role: osd.2 2024-03-25T19:39:25.062 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:39:25.062 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:25.128 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:25.160 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-03-25T19:39:25.163 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-03-25T19:39:25.163 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:25.234 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:25.305 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:25.370 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:25.370 INFO:tasks.ceph:role: osd.3 2024-03-25T19:39:25.371 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:39:25.371 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:39:25.436 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:25.436 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:25.436 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:25.436 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:25.436 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:25.436 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:25.437 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:25.437 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:25.437 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:25.437 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:25.468 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-03-25T19:39:25.470 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-03-25T19:39:25.470 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:25.540 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:25.611 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:25.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:25.706+0000 7fd4fb60c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:39:25.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:25.706+0000 7fd4fb60c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:39:25.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:25.706+0000 7fd4fb60c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:25.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:25.706+0000 7fd4fb60c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:25.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:25.706+0000 7fd4fb60c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:25.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:25.706+0000 7fd4fb60c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:39:28.546 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:28.576 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:28.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:28.667+0000 7fbb6ba11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:39:28.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:28.667+0000 7fbb6ba11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:39:28.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:28.667+0000 7fbb6ba11640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:28.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:28.667+0000 7fbb6ba11640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:28.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:28.667+0000 7fbb6ba11640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:28.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:28.667+0000 7fbb6ba11640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:39:31.507 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:31.538 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:31.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:31.629+0000 7fdb83a0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:39:31.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:31.629+0000 7fdb83a0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:39:31.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:31.629+0000 7fdb83a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:31.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:31.629+0000 7fdb83a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:31.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:31.629+0000 7fdb83a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:31.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:31.630+0000 7fdb83a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:39:34.464 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:34.495 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:34.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:34.585+0000 7f54fee13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:39:34.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:34.586+0000 7f54fee13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:39:34.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:34.586+0000 7f54fee13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:39:34.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:34.586+0000 7f54fee13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:39:34.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:34.586+0000 7f54fee13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:39:34.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:34.586+0000 7f54fee13640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:39:37.423 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:37.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:39:37.486 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:37.486 INFO:tasks.ceph:role: osd.4 2024-03-25T19:39:37.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:39:37.486 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:39:37.555 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:37.556 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:37.588 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-25T19:39:37.590 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-25T19:39:37.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:39:37.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:39:37.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:39:37.802 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:37.802 INFO:tasks.ceph:role: osd.5 2024-03-25T19:39:37.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:39:37.803 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:39:37.868 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:37.869 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:37.901 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-25T19:39:37.903 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-25T19:39:37.903 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:39:37.974 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:39:38.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:39:38.115 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:38.115 INFO:tasks.ceph:role: osd.6 2024-03-25T19:39:38.115 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:39:38.115 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:39:38.181 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:38.181 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:38.181 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:38.181 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:38.182 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:38.182 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:38.182 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:38.182 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:38.182 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:38.182 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:38.215 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-25T19:39:38.217 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-25T19:39:38.217 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:39:38.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:39:38.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:39:38.426 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:38.426 INFO:tasks.ceph:role: osd.7 2024-03-25T19:39:38.426 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:39:38.426 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:39:38.492 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:38.492 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:38.492 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:38.492 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:38.492 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:38.492 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:38.493 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:38.493 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:38.493 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:38.493 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:38.525 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-25T19:39:38.527 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-25T19:39:38.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:39:38.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:39:38.667 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:38.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:38.765+0000 7f49d8215640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:39:38.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:38.765+0000 7f49d8215640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:39:38.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:38.765+0000 7f49d8215640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:39:38.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:38.765+0000 7f49d8215640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:39:38.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:38.765+0000 7f49d8215640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:39:38.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:38.766+0000 7f49d8215640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:39:41.608 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:39:41.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:41.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:41.728+0000 7ffb6c80d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:39:41.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:41.728+0000 7ffb6c80d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:39:41.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:41.728+0000 7ffb6c80d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:39:41.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:41.728+0000 7ffb6c80d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:39:41.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:41.728+0000 7ffb6c80d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:39:41.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:41.729+0000 7ffb6c80d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:39:44.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:39:44.601 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:44.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:44.693+0000 7fd8d820f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:39:44.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:44.694+0000 7fd8d820f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:39:44.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:44.694+0000 7fd8d820f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:39:44.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:44.694+0000 7fd8d820f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:39:44.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:44.694+0000 7fd8d820f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:39:44.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:44.694+0000 7fd8d820f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:39:47.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:39:47.563 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:47.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:47.655+0000 7fa32d815640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:39:47.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:47.655+0000 7fa32d815640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:39:47.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:47.655+0000 7fa32d815640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:39:47.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:47.655+0000 7fa32d815640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:39:47.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:47.655+0000 7fa32d815640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:39:47.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:39:47.655+0000 7fa32d815640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:39:50.496 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:39:50.613 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:39:50.614 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:50.614 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:39:50.645 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:50.645 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:39:50.716 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:50.716 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:39:50.786 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:50.787 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:39:50.856 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:50.856 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:39:50.926 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:50.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:39:50.958 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:50.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:39:51.028 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:51.029 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:39:51.099 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:51.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:39:51.168 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:51.168 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:39:51.237 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:39:51.237 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:39:51.254 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:39:51.254 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:39:51.292 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:39:51.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:39:51.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDs0gFmqRI7DBAAhOzlATB44bOOs63UgqHcXw== 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDt0gFmtT8pKhAAXAiou5ZeoDP+VacMrJAPgQ== 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDw0gFmdODPJxAAgCrLI6IJBtRBJ02syO3i+Q== 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDz0gFmVEGSJRAAd2mOiBADgPo9qjBdZYBi0Q== 2024-03-25T19:39:51.335 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD20gFm5xIJIxAAzYI0biDd4KGYNLbspkkhOw== 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDs0gFmmlzDERAAqQAgtEP4FzMiTE4G3DGJOw== 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD60gFm7OauLRAAerBZLkh7E/506dZnr9uZIg== 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD90gFm+8l6KxAA3pV+4OK6wtK5Hk4NR0kA8w== 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAA0wFmXQ1pKRAANu26cXW56TMYVJjDZtBkog== 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAD0wFmNJ0cJxAAkMqzWGJXrSsDiYQ0GhUlTA== 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDs0gFmay+SFBAAw7ztmS5jNwXjO+EfF8Vy0g== 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2024-03-25T19:39:51.336 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDs0gFmcnD8FxAAJLY29gxX9Z1vqg4DO1E+jA== 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.y] 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDs0gFmqRI7DBAAhOzlATB44bOOs63UgqHcXw== 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDt0gFmtT8pKhAAXAiou5ZeoDP+VacMrJAPgQ== 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDw0gFmdODPJxAAgCrLI6IJBtRBJ02syO3i+Q== 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDz0gFmVEGSJRAAd2mOiBADgPo9qjBdZYBi0Q== 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.3] 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQD20gFm5xIJIxAAzYI0biDd4KGYNLbspkkhOw== 2024-03-25T19:39:51.358 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.x] 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDs0gFmmlzDERAAqQAgtEP4FzMiTE4G3DGJOw== 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.4] 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQD60gFm7OauLRAAerBZLkh7E/506dZnr9uZIg== 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.5] 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQD90gFm+8l6KxAA3pV+4OK6wtK5Hk4NR0kA8w== 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.6] 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAA0wFmXQ1pKRAANu26cXW56TMYVJjDZtBkog== 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.7] 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAD0wFmNJ0cJxAAkMqzWGJXrSsDiYQ0GhUlTA== 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDs0gFmay+SFBAAw7ztmS5jNwXjO+EfF8Vy0g== 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout:[client.1] 2024-03-25T19:39:51.359 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDs0gFmcnD8FxAAJLY29gxX9Z1vqg4DO1E+jA== 2024-03-25T19:39:51.360 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:39:51.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:39:51.452 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.507 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.509 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.562 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.564 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.668 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:39:51.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:39:51.722 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.776 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.777 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.829 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.831 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.882 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:39:51.936 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:39:51.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:39:51.994 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:39:51.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:39:52.051 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:39:52.051 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:39:52.077 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:39:52.173 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:39:52.199 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:39:52.265 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:39:52.363 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:39:52.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:39:52.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:39:52.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:39:52.543 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:52.545 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:52.597 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:39:52.598 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:39:52.598 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:39:52.598 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:39:52.600 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:39:52.600 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:39:52.600 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:39:52.600 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:39:52.602 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:39:52.602 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:39:52.602 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:39:52.603 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:39:52.639 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:39:52.639 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:39:52.640 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:39:52.640 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:39:52.642 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:39:52.642 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:39:52.642 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:39:52.644 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:39:52.644 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:39:52.644 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:39:52.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.746+0000 7fe154ea8640 10 monclient: get_monmap_and_config 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.746+0000 7fe154ea8640 10 monclient: build_initial_monmap 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.746+0000 7fe154ea8640 10 monclient: monmap: 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:52.747097+0000 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:52.747097+0000 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:39:52.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.746+0000 7fe154ea8640 1 Processor -- start 2024-03-25T19:39:52.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.746+0000 7fe154ea8640 1 -- start start 2024-03-25T19:39:52.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.746+0000 7fe154ea8640 10 monclient: init 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fe1500a9920 addr v1:172.21.15.110:6790/0 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fe1500b45a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fe150146450 addr v1:172.21.15.178:6789/0 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15006b100 con 0x7fe1500b45a0 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500b2940 con 0x7fe150146450 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15013fec0 con 0x7fe1500a9920 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:52.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe154ea8640 10 monclient(hunting): authenticate will time out at 4163.888415s 2024-03-25T19:39:52.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe14e59c640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe15013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:52.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe14dd9b640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe150145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:52.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.747+0000 7fe14ed9d640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe15014dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:52.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.948+0000 7fe14dd9b640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe150145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:52.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.948+0000 7fe14e59c640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe15013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:52.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:52.948+0000 7fe14ed9d640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe15014dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:53.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:53.349+0000 7fe14dd9b640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe150145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:53.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:53.349+0000 7fe14e59c640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe15013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:53.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:53.349+0000 7fe14ed9d640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe15014dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:54.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:54.150+0000 7fe14dd9b640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe150145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:54.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:54.150+0000 7fe14e59c640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe15013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:54.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:54.150+0000 7fe14ed9d640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe15014dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:39:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:39:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:39:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:39:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe15013f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe15014dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe150145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe150146450 addr v1:172.21.15.178:6789/0 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe1500a9920 addr v1:172.21.15.110:6790/0 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe1500b45a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15006b100 con 0x7fe1500b45a0 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500b2940 con 0x7fe150146450 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15013fec0 con 0x7fe1500a9920 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe14dd9b640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe130008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe14e59c640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.747+0000 7fe14ed9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe13000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.948+0000 7fe14ed9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe13000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.948+0000 7fe14dd9b640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe130008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:55.948+0000 7fe14e59c640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:56.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:56.349+0000 7fe14ed9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe13000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:56.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:56.349+0000 7fe14dd9b640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe130008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:56.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:56.349+0000 7fe14e59c640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:57.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:57.150+0000 7fe14ed9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe13000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:57.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:57.150+0000 7fe14dd9b640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe130008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:57.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:57.150+0000 7fe14e59c640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:58.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe130008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe13000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe1500a9920 addr v1:172.21.15.110:6790/0 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe150146450 addr v1:172.21.15.178:6789/0 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe1500b45a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130010e20 con 0x7fe1500b45a0 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15006b100 con 0x7fe150146450 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500b2940 con 0x7fe1500a9920 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:58.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe14e59c640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe13000fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:58.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.747+0000 7fe14ed9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe1300105c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:58.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.748+0000 7fe14dd9b640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:58.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.949+0000 7fe14e59c640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe13000fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:58.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.949+0000 7fe14ed9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe1300105c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:58.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:58.949+0000 7fe14dd9b640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:59.350+0000 7fe14e59c640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe13000fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:39:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:59.350+0000 7fe14ed9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe1300105c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:39:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:39:59.350+0000 7fe14dd9b640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:39:59.591 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:00:06.846 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.835 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:00:07.120 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.838 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:00:07.122 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.897 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:00:07.152 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:00.145 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:00:07.429 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:00.148 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:00:07.432 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:00.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:00.151+0000 7fe14e59c640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe13000fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6790/0 2024-03-25T19:40:00.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:00.151+0000 7fe14ed9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe1300105c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-03-25T19:40:00.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:00.151+0000 7fe14dd9b640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.178:6789/0 2024-03-25T19:40:01.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 1 -- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe13000fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 1 -- >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe1300105c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe1500a9920 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe1500b45a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe150146450 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15013fec0 con 0x7fe1500b45a0 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130010e20 con 0x7fe150146450 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15006b100 con 0x7fe1500a9920 2024-03-25T19:40:01.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.748+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:01.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.753+0000 7fe14e59c640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 0x7fe13000fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:37144/0 (socket says 172.21.15.110:37144) 2024-03-25T19:40:01.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:01.753+0000 7fe14e59c640 1 -- 172.21.15.110:0/1246391 learned_addr learned my addr 172.21.15.110:0/1246391 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe1500a9920 legacy=0x7fe13000fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.178:6789/0 conn(0x7fe150146450 legacy=0x7fe1300056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe130010580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe13000bf70 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe130012080 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe1500b45a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:04.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500b2940 con 0x7fe1500b45a0 2024-03-25T19:40:04.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130005730 con 0x7fe13000bf70 2024-03-25T19:40:04.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130005960 con 0x7fe130012080 2024-03-25T19:40:04.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:04.748+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:07.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:40:07.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:07.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:07.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:07.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe130014470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:07.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.178:6789/0 conn(0x7fe13000bf70 legacy=0x7fe13000c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:07.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe1500b45a0 legacy=0x7fe130010580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:07.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:07.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe130005e30 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:07.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe130012080 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:07.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe13000bf70 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:07.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130005b90 con 0x7fe130005e30 2024-03-25T19:40:07.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130006ac0 con 0x7fe13000bf70 2024-03-25T19:40:07.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130010580 con 0x7fe130012080 2024-03-25T19:40:07.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:07.748+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:10.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe13000fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.178:6789/0 conn(0x7fe13000bf70 legacy=0x7fe13000c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe130005e30 legacy=0x7fe130006210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe1300108a0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe130012080 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe13000c3e0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130010710 con 0x7fe13000c3e0 2024-03-25T19:40:10.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13000c890 con 0x7fe1300108a0 2024-03-25T19:40:10.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130006210 con 0x7fe130012080 2024-03-25T19:40:10.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:10.749+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe13000fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.178:6789/0 conn(0x7fe1300108a0 legacy=0x7fe130006e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe13000c3e0 legacy=0x7fe130007690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe13000cda0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe130012080 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe13000ffb0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:13.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130006440 con 0x7fe13000cda0 2024-03-25T19:40:13.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1300066c0 con 0x7fe13000ffb0 2024-03-25T19:40:13.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130007690 con 0x7fe130012080 2024-03-25T19:40:13.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:13.749+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:40:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe13000f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.178:6789/0 conn(0x7fe13000ffb0 legacy=0x7fe130006e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe13000cda0 legacy=0x7fe13000d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe1300079b0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe130012080 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe13000ffb0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130007820 con 0x7fe1300079b0 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13000d180 con 0x7fe13000ffb0 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13000d310 con 0x7fe130012080 2024-03-25T19:40:16.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.750+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.752+0000 7fe14dd9b640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe13000f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T19:40:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.752+0000 7fe14dd9b640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe13000f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:40:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.752+0000 7fe14dd9b640 1 --1- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 0x7fe13000f8a0 :-1 s=OPENED pgs=25 cs=1 l=1).handle_message read tag failed 2024-03-25T19:40:16.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.752+0000 7fe14dd9b640 1 --1- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 0x7fe13000f8a0 :-1 s=OPENED pgs=25 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:40:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.752+0000 7fe14d59a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.110:6790/0 2024-03-25T19:40:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.753+0000 7fe14e59c640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe1300079b0 legacy=0x7fe130011070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:40:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.753+0000 7fe14e59c640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe1300079b0 legacy=0x7fe130011070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:40:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.753+0000 7fe14e59c640 1 --1- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe1300079b0 0x7fe130011070 :-1 s=OPENED pgs=24 cs=1 l=1).handle_message read tag failed 2024-03-25T19:40:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.753+0000 7fe14e59c640 1 --1- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe1300079b0 0x7fe130011070 :-1 s=OPENED pgs=24 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:40:16.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:16.753+0000 7fe14d59a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.110:6789/0 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 10 monclient(hunting): tick 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe13000f8a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.178:6789/0 conn(0x7fe13000ffb0 legacy=0x7fe130006e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe1300079b0 legacy=0x7fe130011070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe130012080 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe130011070 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:19.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe13000ffb0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:19.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13000d4f0 con 0x7fe130011070 2024-03-25T19:40:19.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13000fdb0 con 0x7fe13000ffb0 2024-03-25T19:40:19.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe130011450 con 0x7fe130012080 2024-03-25T19:40:19.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:19.749+0000 7fe13ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:20.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.713+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096663368 0 0) 0x7fe13000d4f0 con 0x7fe130011070 2024-03-25T19:40:20.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.713+0000 7fe14d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.713+0000 7fe14d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.713+0000 7fe14d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.713+0000 7fe14d59a640 10 monclient(hunting): my global_id is 4098 2024-03-25T19:40:20.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.713+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe138003520 con 0x7fe130011070 2024-03-25T19:40:20.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.729+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644186180 0 0) 0x7fe13000fdb0 con 0x7fe13000ffb0 2024-03-25T19:40:20.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.729+0000 7fe14d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.729+0000 7fe14d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.729+0000 7fe14d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.729+0000 7fe14d59a640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:40:20.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.729+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe13000d4f0 con 0x7fe13000ffb0 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204391151 0 0) 0x7fe138003520 con 0x7fe130011070 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe130011820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.178:6789/0 conn(0x7fe13000ffb0 legacy=0x7fe130006e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 1 monclient: found mon.noname-a 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1300115e0 con 0x7fe130011070 2024-03-25T19:40:20.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 10 monclient: _finish_auth 0 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe144004120 con 0x7fe130011070 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe154ea8640 5 monclient: authenticate success, global_id 4098 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe154ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe14d59a640 10 monclient: dump: 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:20.771 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:20.772 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:20.772 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:20.772 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:20.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.770+0000 7fe154ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:20.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe144004280 con 0x7fe130011070 2024-03-25T19:40:20.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe14d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:20.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe14d59a640 1 -- 172.21.15.110:0/1246391 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe1440044f0 con 0x7fe130011070 2024-03-25T19:40:20.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe14d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe14d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe14d59a640 10 monclient: dump: 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe154ea8640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:20.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe154ea8640 4 set_mon_vals no callback set 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe154ea8640 10 monclient: shutdown 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe154ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.807+0000 7fe154ea8640 1 -- 172.21.15.110:0/1246391 >> v1:172.21.15.110:6789/0 conn(0x7fe130011070 legacy=0x7fe13000d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.808+0000 7fe154ea8640 1 -- 172.21.15.110:0/1246391 shutdown_connections 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.808+0000 7fe154ea8640 1 -- 172.21.15.110:0/1246391 >> 172.21.15.110:0/1246391 conn(0x7fe1500b7f40 msgr2=0x7fe1500b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.808+0000 7fe154ea8640 1 -- 172.21.15.110:0/1246391 shutdown_connections 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.808+0000 7fe154ea8640 1 -- 172.21.15.110:0/1246391 wait complete. 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 10 monclient: build_initial_monmap 2024-03-25T19:40:20.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 10 monclient: monmap: 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 librados: starting msgr at 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 librados: starting objecter 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 Processor -- start 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 -- start start 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 librados: setting wanted keys 2024-03-25T19:40:20.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 librados: calling monclient init 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 10 monclient: init 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fe13000ffb0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fe130011070 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fe130012080 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500adab0 con 0x7fe130011070 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500ac020 con 0x7fe13000ffb0 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.809+0000 7fe154ea8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe150138ac0 con 0x7fe130012080 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.810+0000 7fe154ea8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:20.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.810+0000 7fe154ea8640 10 monclient(hunting): authenticate will time out at 4191.950837s 2024-03-25T19:40:20.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.811+0000 7fe14ed9d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 0x7fe1501383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41622/0 (socket says 172.21.15.110:41622) 2024-03-25T19:40:20.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.811+0000 7fe14ed9d640 1 -- 172.21.15.110:0/3973417710 learned_addr learned my addr 172.21.15.110:0/3973417710 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:20.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.812+0000 7fe13effd640 10 client.?.objecter ms_handle_connect 0x7fe130012080 2024-03-25T19:40:20.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.812+0000 7fe13effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:20.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.816+0000 7fe13effd640 10 client.?.objecter ms_handle_connect 0x7fe13000ffb0 2024-03-25T19:40:20.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.816+0000 7fe13effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:20.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.854+0000 7fe13effd640 10 client.?.objecter ms_handle_connect 0x7fe130011070 2024-03-25T19:40:20.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.854+0000 7fe13effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:20.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.865+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340425739 0 0) 0x7fe1500adab0 con 0x7fe130011070 2024-03-25T19:40:20.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.865+0000 7fe13effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.865+0000 7fe13effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.865+0000 7fe13effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.865+0000 7fe13effd640 10 monclient(hunting): my global_id is 4107 2024-03-25T19:40:20.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.865+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe130011da0 con 0x7fe130011070 2024-03-25T19:40:20.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.886+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182672285 0 0) 0x7fe130011450 con 0x7fe13000ffb0 2024-03-25T19:40:20.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.886+0000 7fe13effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.886+0000 7fe13effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.886+0000 7fe13effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.886+0000 7fe13effd640 10 monclient(hunting): my global_id is 4108 2024-03-25T19:40:20.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.886+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1500adab0 con 0x7fe13000ffb0 2024-03-25T19:40:20.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.887+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724115972 0 0) 0x7fe130011da0 con 0x7fe130011070 2024-03-25T19:40:20.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.887+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe130011450 con 0x7fe130011070 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.887+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe1440045d0 con 0x7fe130011070 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.887+0000 7fe13effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.887+0000 7fe13effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.887+0000 7fe13effd640 10 monclient(hunting): dump: 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:20.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.887+0000 7fe13effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:20.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.888+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844050179 0 0) 0x7fe13000d4f0 con 0x7fe130012080 2024-03-25T19:40:20.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.888+0000 7fe13effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.888+0000 7fe13effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.888+0000 7fe13effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.888+0000 7fe13effd640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:40:20.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.888+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe130011da0 con 0x7fe130012080 2024-03-25T19:40:20.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 297718634 0 0) 0x7fe130011450 con 0x7fe130011070 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 >> v1:172.21.15.110:6790/0 conn(0x7fe130012080 legacy=0x7fe1501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 >> v1:172.21.15.178:6789/0 conn(0x7fe13000ffb0 legacy=0x7fe1500ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 1 monclient: found mon.a 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1500b1bf0 con 0x7fe130011070 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 10 monclient: _finish_auth 0 2024-03-25T19:40:21.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe13effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 5 monclient: authenticate success, global_id 4107 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 10 monclient: _renew_subs 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1500ac020 con 0x7fe130011070 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 10 monclient: _renew_subs 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe150138ac0 con 0x7fe130011070 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.900+0000 7fe154ea8640 1 librados: init done 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.901+0000 7fe154ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.901+0000 7fe154ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.901+0000 7fe154ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:21.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.901+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1500ac020 con 0x7fe130011070 2024-03-25T19:40:21.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.954+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1440049e0 con 0x7fe130011070 2024-03-25T19:40:21.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.954+0000 7fe13effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:21.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.954+0000 7fe14cd99640 4 set_mon_vals no callback set 2024-03-25T19:40:21.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.954+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe144004df0 con 0x7fe130011070 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.954+0000 7fe13effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.954+0000 7fe13effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:20.954+0000 7fe13effd640 10 monclient: dump: 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:21.021 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:21.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe144005360 con 0x7fe130011070 2024-03-25T19:40:21.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 319419998 0 0) 0x7fe1440065e0 con 0x7fe130011070 2024-03-25T19:40:21.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 10 client.4107.objecter ms_dispatch 0x7fe150000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:40:21.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:40:21.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:40:21.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T19:40:21.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe144018dc0 con 0x7fe130011070 2024-03-25T19:40:21.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:21.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.158+0000 7fe13effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:21.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.244+0000 7fe154ea8640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:21.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.244+0000 7fe154ea8640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:21.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.244+0000 7fe154ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:21.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.245+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe1500ac020 con 0x7fe130011070 2024-03-25T19:40:21.381 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:21.380+0000 7fb51062b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:21.436 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:21.436+0000 7fb51062b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:21.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.497+0000 7fe13effd640 1 -- 172.21.15.110:0/3973417710 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fe1440056f0 con 0x7fe130011070 2024-03-25T19:40:21.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.497+0000 7fe13effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:21.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.497+0000 7fe13effd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:40:21.496 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 10 librados: watch_flush enter 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 10 librados: watch_flush exit 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 10 client.4107.objecter successfully canceled tick 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 10 monclient: shutdown 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 >> v1:172.21.15.110:6789/0 conn(0x7fe130011070 legacy=0x7fe150137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 shutdown_connections 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 >> 172.21.15.110:0/3973417710 conn(0x7fe1500b7f40 msgr2=0x7fe1500b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 shutdown_connections 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 1 -- 172.21.15.110:0/3973417710 wait complete. 2024-03-25T19:40:21.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:21.498+0000 7fe154ea8640 1 librados: shutdown 2024-03-25T19:40:21.508 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:40:21.509 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:40:21.509 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:40:21.509 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:40:21.538 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:40:21.538 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:40:21.605 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:40:21.605 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:40:21.673 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-03-25T19:40:21.673 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:40:21.702 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:21.702+0000 7f4bc9813040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:21.729 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:40:21.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:40:21.756 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:40:21.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:40:21.765 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:21.766+0000 7f4bc9813040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:21.823 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:40:21.823 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:40:21.890 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-25T19:40:21.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:40:21.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 048b4cae-9f46-46e8-a412-3809fe6e4b7b 0 2024-03-25T19:40:21.998 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:21.998+0000 7fb51062b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:22.056 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.056+0000 7fb51062b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.098+0000 7f284261e640 10 monclient: get_monmap_and_config 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.098+0000 7f284261e640 10 monclient: build_initial_monmap 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.098+0000 7f284261e640 10 monclient: monmap: 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:40:22.099829+0000 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:40:22.099829+0000 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:22.098 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.099 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:22.099 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:22.099 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:22.099 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.098+0000 7f284261e640 1 Processor -- start 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.098+0000 7f284261e640 1 -- start start 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.098+0000 7f284261e640 10 monclient: init 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 10 monclient(hunting): picked mon.noname-b con 0x7f283c0b4700 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 10 monclient(hunting): picked mon.noname-a con 0x7f283c142a20 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 10 monclient(hunting): picked mon.noname-c con 0x7f283c0a96b0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c0a9d60 con 0x7f283c142a20 2024-03-25T19:40:22.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c13ff10 con 0x7f283c0a96b0 2024-03-25T19:40:22.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c0aa1f0 con 0x7f283c0b4700 2024-03-25T19:40:22.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:22.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.099+0000 7f284261e640 10 monclient(hunting): authenticate will time out at 4177.249899s 2024-03-25T19:40:22.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.100+0000 7f2840bbb640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f283c0a96b0 0x7f283c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34234/0 (socket says 172.21.15.178:34234) 2024-03-25T19:40:22.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.100+0000 7f2840bbb640 1 -- 172.21.15.178:0/3478918397 learned_addr learned my addr 172.21.15.178:0/3478918397 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:22.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.102+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264822054 0 0) 0x7f283c13ff10 con 0x7f283c0a96b0 2024-03-25T19:40:22.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.102+0000 7f283affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.102+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.102+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.102+0000 7f283affd640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:40:22.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.102+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2824003520 con 0x7f283c0a96b0 2024-03-25T19:40:22.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364099415 0 0) 0x7f283c0aa1f0 con 0x7f283c0b4700 2024-03-25T19:40:22.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:40:22.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c13ff10 con 0x7f283c0b4700 2024-03-25T19:40:22.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911690165 0 0) 0x7f283c0a9d60 con 0x7f283c142a20 2024-03-25T19:40:22.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:40:22.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.103+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c0aa1f0 con 0x7f283c142a20 2024-03-25T19:40:22.104 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.103+0000 7fb51062b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188683405 0 0) 0x7f2824003520 con 0x7f283c0a96b0 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 >> v1:172.21.15.110:6790/0 conn(0x7f283c0b4700 legacy=0x7f283c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 >> v1:172.21.15.110:6789/0 conn(0x7f283c142a20 legacy=0x7f283c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 1 monclient: found mon.noname-c 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c0b4ae0 con 0x7f283c0a96b0 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 10 monclient: _finish_auth 0 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2834001a10 con 0x7f283c0a96b0 2024-03-25T19:40:22.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f284261e640 5 monclient: authenticate success, global_id 4117 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f284261e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f283affd640 10 monclient: dump: 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.105+0000 7f284261e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:22.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.107+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2834001b70 con 0x7f283c0a96b0 2024-03-25T19:40:22.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.107+0000 7f283affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:22.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.107+0000 7f284261e640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:22.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.107+0000 7f284261e640 4 set_mon_vals no callback set 2024-03-25T19:40:22.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f283affd640 1 -- 172.21.15.178:0/3478918397 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2834001f80 con 0x7f283c0a96b0 2024-03-25T19:40:22.109 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f283affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f283affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f283affd640 10 monclient: dump: 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f284261e640 10 monclient: shutdown 2024-03-25T19:40:22.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f284261e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f284261e640 1 -- 172.21.15.178:0/3478918397 >> v1:172.21.15.178:6789/0 conn(0x7f283c0a96b0 legacy=0x7f283c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f284261e640 1 -- 172.21.15.178:0/3478918397 shutdown_connections 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f284261e640 1 -- 172.21.15.178:0/3478918397 >> 172.21.15.178:0/3478918397 conn(0x7f283c0668e0 msgr2=0x7f283c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f284261e640 1 -- 172.21.15.178:0/3478918397 shutdown_connections 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.108+0000 7f284261e640 1 -- 172.21.15.178:0/3478918397 wait complete. 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.109+0000 7f284261e640 10 monclient: build_initial_monmap 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.109+0000 7f284261e640 10 monclient: monmap: 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:22.111 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:22.112 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.109+0000 7f284261e640 1 librados: starting msgr at 2024-03-25T19:40:22.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.109+0000 7f284261e640 1 librados: starting objecter 2024-03-25T19:40:22.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.109+0000 7f284261e640 1 Processor -- start 2024-03-25T19:40:22.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 1 -- start start 2024-03-25T19:40:22.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 1 librados: setting wanted keys 2024-03-25T19:40:22.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 1 librados: calling monclient init 2024-03-25T19:40:22.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 10 monclient: init 2024-03-25T19:40:22.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:22.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 10 monclient(hunting): picked mon.noname-a con 0x7f283c0a96b0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 10 monclient(hunting): picked mon.noname-c con 0x7f283c0b4700 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 10 monclient(hunting): picked mon.noname-b con 0x7f283c142a20 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c1d76b0 con 0x7f283c0a96b0 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c1d7380 con 0x7f283c142a20 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c1d4fe0 con 0x7f283c0b4700 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.110+0000 7f284261e640 10 monclient(hunting): authenticate will time out at 4177.261171s 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.111+0000 7f2840bbb640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f283c142a20 0x7f283c1e1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34236/0 (socket says 172.21.15.178:34236) 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.111+0000 7f2840bbb640 1 -- 172.21.15.178:0/3697497796 learned_addr learned my addr 172.21.15.178:0/3697497796 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.111+0000 7f2838ff9640 10 client.?.objecter ms_handle_connect 0x7f283c142a20 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.111+0000 7f2838ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.111+0000 7f2838ff9640 10 client.?.objecter ms_handle_connect 0x7f283c0a96b0 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.111+0000 7f2838ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.111+0000 7f2838ff9640 10 client.?.objecter ms_handle_connect 0x7f283c0b4700 2024-03-25T19:40:22.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.111+0000 7f2838ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413718225 0 0) 0x7f283c1d7380 con 0x7f283c142a20 2024-03-25T19:40:22.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:40:22.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2828004eb0 con 0x7f283c142a20 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994682578 0 0) 0x7f283c1d4fe0 con 0x7f283c0b4700 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c1d7380 con 0x7f283c0b4700 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925221849 0 0) 0x7f283c1d76b0 con 0x7f283c0a96b0 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:40:22.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.113+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c1d4fe0 con 0x7f283c0a96b0 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.117+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1356720031 0 0) 0x7f2828004eb0 con 0x7f283c142a20 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.117+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f283c1d76b0 con 0x7f283c142a20 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.117+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2834001e80 con 0x7f283c142a20 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.118+0000 7f2838ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.118+0000 7f2838ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.118+0000 7f2838ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.118 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.119 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.119 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.119 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.119 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.119 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.119 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.119 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.118+0000 7f2838ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:22.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 70368521 0 0) 0x7f283c1d4fe0 con 0x7f283c0a96b0 2024-03-25T19:40:22.120 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2828004eb0 con 0x7f283c0a96b0 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2830001930 con 0x7f283c0a96b0 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4119593855 0 0) 0x7f283c1d7380 con 0x7f283c0b4700 2024-03-25T19:40:22.121 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f283c1d4fe0 con 0x7f283c0b4700 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f282c0017d0 con 0x7f283c0b4700 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.120+0000 7f2838ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.122 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2547142276 0 0) 0x7f2828004eb0 con 0x7f283c0a96b0 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 >> v1:172.21.15.110:6790/0 conn(0x7f283c0b4700 legacy=0x7f283c13bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 >> v1:172.21.15.178:6789/0 conn(0x7f283c142a20 legacy=0x7f283c1e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 1 monclient: found mon.a 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c1d8250 con 0x7f283c0a96b0 2024-03-25T19:40:22.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 10 monclient: _finish_auth 0 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f2838ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 10 monclient: _renew_subs 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c13c500 con 0x7f283c0a96b0 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 10 monclient: _renew_subs 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c13b7c0 con 0x7f283c0a96b0 2024-03-25T19:40:22.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.124+0000 7f284261e640 1 librados: init done 2024-03-25T19:40:22.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.125+0000 7f284261e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:22.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.125+0000 7f284261e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:22.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.125+0000 7f284261e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:22.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.125+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c04f730 con 0x7f283c0a96b0 2024-03-25T19:40:22.160 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.160+0000 7fb51062b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:22.202 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.201+0000 7fb51062b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:22.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2830000f90 con 0x7f283c0a96b0 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2830003790 con 0x7f283c0a96b0 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 10 monclient: dump: 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.225 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.226 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2830003c80 con 0x7f283c0a96b0 2024-03-25T19:40:22.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f283a7fc640 4 set_mon_vals no callback set 2024-03-25T19:40:22.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1923985864 0 0) 0x7f2830005160 con 0x7f283c0a96b0 2024-03-25T19:40:22.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 10 client.4119.objecter ms_dispatch 0x7f283c000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.224+0000 7f2838ff9640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.225+0000 7f2838ff9640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.225+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f28300041d0 con 0x7f283c0a96b0 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.225+0000 7f2838ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:22.227 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.225+0000 7f2838ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:22.297 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.297+0000 7f284261e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "048b4cae-9f46-46e8-a412-3809fe6e4b7b", "id": 0}] 2024-03-25T19:40:22.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.297+0000 7f284261e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "048b4cae-9f46-46e8-a412-3809fe6e4b7b", "id": 0}] 2024-03-25T19:40:22.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.297+0000 7f284261e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:22.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.297+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "048b4cae-9f46-46e8-a412-3809fe6e4b7b", "id": 0} v 0) v1 -- 0x7f283c069f30 con 0x7f283c0a96b0 2024-03-25T19:40:22.320 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.320+0000 7fb51062b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:22.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.338+0000 7f4bc9813040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:22.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.383+0000 7f2838ff9640 1 -- 172.21.15.178:0/3697497796 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "048b4cae-9f46-46e8-a412-3809fe6e4b7b", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3713261539 0 732707515) 0x7f28300043f0 con 0x7f283c0a96b0 2024-03-25T19:40:22.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.383+0000 7f2838ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "048b4cae-9f46-46e8-a412-3809fe6e4b7b", "id": 0}] 2024-03-25T19:40:22.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.383+0000 7f2838ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:22.384 INFO:teuthology.orchestra.run.smithi178.stdout:0 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 10 librados: watch_flush enter 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 10 librados: watch_flush exit 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 10 monclient: shutdown 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 >> v1:172.21.15.110:6789/0 conn(0x7f283c0a96b0 legacy=0x7f283c1d7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 shutdown_connections 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 >> 172.21.15.178:0/3697497796 conn(0x7f283c0668e0 msgr2=0x7f283c0b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 shutdown_connections 2024-03-25T19:40:22.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 1 -- 172.21.15.178:0/3697497796 wait complete. 2024-03-25T19:40:22.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.384+0000 7f284261e640 1 librados: shutdown 2024-03-25T19:40:22.391 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.390+0000 7fb51062b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:22.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.396+0000 7f4bc9813040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:22.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new d209ed26-5603-4a13-a52f-7a40169d3c6a 1 2024-03-25T19:40:22.440 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.440+0000 7fb51062b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:22.444 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.445+0000 7f4bc9813040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:22.496 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.495+0000 7fb51062b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:22.498 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.498+0000 7ff1cc295640 10 monclient: get_monmap_and_config 2024-03-25T19:40:22.498 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.498+0000 7ff1cc295640 10 monclient: build_initial_monmap 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.498+0000 7ff1cc295640 10 monclient: monmap: 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:40:22.500299+0000 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:40:22.500299+0000 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:22.499 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.500 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.500+0000 7f4bc9813040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 1 Processor -- start 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 1 -- start start 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 10 monclient: init 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 10 monclient(hunting): picked mon.noname-b con 0x7ff1c40b4700 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 10 monclient(hunting): picked mon.noname-a con 0x7ff1c4142a20 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 10 monclient(hunting): picked mon.noname-c con 0x7ff1c40a96b0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40a9d60 con 0x7ff1c4142a20 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c413ff10 con 0x7ff1c40a96b0 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40aa1f0 con 0x7ff1c40b4700 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:22.501 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.499+0000 7ff1cc295640 10 monclient(hunting): authenticate will time out at 4177.650383s 2024-03-25T19:40:22.502 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.500+0000 7ff1ca832640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff1c40a96b0 0x7ff1c413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34248/0 (socket says 172.21.15.178:34248) 2024-03-25T19:40:22.502 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.500+0000 7ff1ca832640 1 -- 172.21.15.178:0/3451876049 learned_addr learned my addr 172.21.15.178:0/3451876049 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393558420 0 0) 0x7ff1c413ff10 con 0x7ff1c40a96b0 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b0003520 con 0x7ff1c40a96b0 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460715438 0 0) 0x7ff1c40aa1f0 con 0x7ff1c40b4700 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.504 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:40:22.504 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.502+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c413ff10 con 0x7ff1c40b4700 2024-03-25T19:40:22.504 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.503+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344465555 0 0) 0x7ff1c40a9d60 con 0x7ff1c4142a20 2024-03-25T19:40:22.504 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.503+0000 7ff1c902f640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.504 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.503+0000 7ff1c902f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.504 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.503+0000 7ff1c902f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.504 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.503+0000 7ff1c902f640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:40:22.504 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.503+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c40aa1f0 con 0x7ff1c4142a20 2024-03-25T19:40:22.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1483624164 0 0) 0x7ff1b0003520 con 0x7ff1c40a96b0 2024-03-25T19:40:22.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 >> v1:172.21.15.110:6790/0 conn(0x7ff1c40b4700 legacy=0x7ff1c4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 >> v1:172.21.15.110:6789/0 conn(0x7ff1c4142a20 legacy=0x7ff1c413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 10 monclient: _finish_hunting 0 2024-03-25T19:40:22.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 1 monclient: found mon.noname-c 2024-03-25T19:40:22.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:22.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:40:22.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c40b4ae0 con 0x7ff1c40a96b0 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 10 monclient: _finish_auth 0 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 10 monclient: _check_auth_tickets 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff1bc001a10 con 0x7ff1c40a96b0 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1cc295640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1cc295640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1c902f640 10 monclient: dump: 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.507 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.508 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.508 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.508 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.508 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.508 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.505+0000 7ff1cc295640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:22.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.507+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1bc001b70 con 0x7ff1c40a96b0 2024-03-25T19:40:22.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.507+0000 7ff1c902f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:22.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.507+0000 7ff1cc295640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:22.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.507+0000 7ff1cc295640 4 set_mon_vals no callback set 2024-03-25T19:40:22.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.507+0000 7ff1c902f640 1 -- 172.21.15.178:0/3451876049 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff1bc001f80 con 0x7ff1c40a96b0 2024-03-25T19:40:22.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.507+0000 7ff1c902f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.507+0000 7ff1c902f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.507+0000 7ff1c902f640 10 monclient: dump: 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.508+0000 7ff1cc295640 10 monclient: shutdown 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.508+0000 7ff1cc295640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:22.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.508+0000 7ff1cc295640 1 -- 172.21.15.178:0/3451876049 >> v1:172.21.15.178:6789/0 conn(0x7ff1c40a96b0 legacy=0x7ff1c413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.508+0000 7ff1cc295640 1 -- 172.21.15.178:0/3451876049 shutdown_connections 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.508+0000 7ff1cc295640 1 -- 172.21.15.178:0/3451876049 >> 172.21.15.178:0/3451876049 conn(0x7ff1c40668e0 msgr2=0x7ff1c4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.508+0000 7ff1cc295640 1 -- 172.21.15.178:0/3451876049 shutdown_connections 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.508+0000 7ff1cc295640 1 -- 172.21.15.178:0/3451876049 wait complete. 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 10 monclient: build_initial_monmap 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 10 monclient: monmap: 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:22.513 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 1 librados: starting msgr at 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 1 librados: starting objecter 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 1 Processor -- start 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 1 -- start start 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 1 librados: setting wanted keys 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 1 librados: calling monclient init 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 10 monclient: init 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.509+0000 7ff1cc295640 10 monclient(hunting): picked mon.noname-c con 0x7ff1c40a96b0 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1cc295640 10 monclient(hunting): picked mon.noname-a con 0x7ff1c40b4700 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1cc295640 10 monclient(hunting): picked mon.noname-b con 0x7ff1c4142a20 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:22.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1cc295640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40ac4a0 con 0x7ff1c40b4700 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1cc295640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40ac780 con 0x7ff1c4142a20 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1cc295640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c413b6e0 con 0x7ff1c40a96b0 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1cc295640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1cc295640 10 monclient(hunting): authenticate will time out at 4177.660568s 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1ca832640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff1c4142a20 0x7ff1c413c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34260/0 (socket says 172.21.15.178:34260) 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1ca832640 1 -- 172.21.15.178:0/1820815899 learned_addr learned my addr 172.21.15.178:0/1820815899 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1baffd640 10 client.?.objecter ms_handle_connect 0x7ff1c4142a20 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1baffd640 10 client.?.objecter ms_handle_connect 0x7ff1c40b4700 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1baffd640 10 client.?.objecter ms_handle_connect 0x7ff1c40a96b0 2024-03-25T19:40:22.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.510+0000 7ff1baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886858100 0 0) 0x7ff1c40ac780 con 0x7ff1c4142a20 2024-03-25T19:40:22.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a8004eb0 con 0x7ff1c4142a20 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967115689 0 0) 0x7ff1c413b6e0 con 0x7ff1c40a96b0 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.512+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c40ac780 con 0x7ff1c40a96b0 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.513+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064169842 0 0) 0x7ff1c40ac4a0 con 0x7ff1c40b4700 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.513+0000 7ff1baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.513+0000 7ff1baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.513+0000 7ff1baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.513+0000 7ff1baffd640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:40:22.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.513+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c413b6e0 con 0x7ff1c40b4700 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227177199 0 0) 0x7ff1a8004eb0 con 0x7ff1c4142a20 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1c40ac4a0 con 0x7ff1c4142a20 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff1bc001e80 con 0x7ff1c4142a20 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 10 monclient(hunting): dump: 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.519 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3176155792 0 0) 0x7ff1c40ac780 con 0x7ff1c40a96b0 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1a8004eb0 con 0x7ff1c40a96b0 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff1c0001970 con 0x7ff1c40a96b0 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.517+0000 7ff1baffd640 10 monclient(hunting): dump: 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.520 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.521 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.521 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.521 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.521 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.521 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.521 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 791856389 0 0) 0x7ff1c413b6e0 con 0x7ff1c40b4700 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1c40ac780 con 0x7ff1c40b4700 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff1b4001790 con 0x7ff1c40b4700 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 10 monclient(hunting): dump: 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.523 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1505711054 0 0) 0x7ff1c40ac4a0 con 0x7ff1c4142a20 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 >> v1:172.21.15.110:6790/0 conn(0x7ff1c40a96b0 legacy=0x7ff1c40acc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 >> v1:172.21.15.110:6789/0 conn(0x7ff1c40b4700 legacy=0x7ff1c4139da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 1 monclient: found mon.b 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c413cb40 con 0x7ff1c4142a20 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:22.524 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 5 monclient: authenticate success, global_id 4126 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 10 monclient: _renew_subs 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c413c860 con 0x7ff1c4142a20 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 10 monclient: _renew_subs 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c41e3480 con 0x7ff1c4142a20 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.520+0000 7ff1cc295640 1 librados: init done 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.521+0000 7ff1cc295640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.521+0000 7ff1cc295640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.521+0000 7ff1cc295640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:22.525 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.521+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c40b1dc0 con 0x7ff1c4142a20 2024-03-25T19:40:22.546 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.546+0000 7fb51062b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:22.550 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.551+0000 7f4bc9813040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:22.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.668+0000 7f4bc9813040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:22.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1bc004e10 con 0x7ff1c4142a20 2024-03-25T19:40:22.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:22.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff1bc005120 con 0x7ff1c4142a20 2024-03-25T19:40:22.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:22.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 10 monclient: dump: 2024-03-25T19:40:22.715 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:22.715 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:22.715 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff1bc005610 con 0x7ff1c4142a20 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1c882e640 4 set_mon_vals no callback set 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1448975571 0 0) 0x7ff1bc006a30 con 0x7ff1c4142a20 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 10 client.4126.objecter ms_dispatch 0x7ff1c4000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:40:22.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T19:40:22.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff1bc018dc0 con 0x7ff1c4142a20 2024-03-25T19:40:22.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:22.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.713+0000 7ff1baffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:22.737 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.738+0000 7f4bc9813040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:22.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.784+0000 7f4bc9813040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:22.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.786+0000 7ff1cc295640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d209ed26-5603-4a13-a52f-7a40169d3c6a", "id": 1}] 2024-03-25T19:40:22.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.786+0000 7ff1cc295640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d209ed26-5603-4a13-a52f-7a40169d3c6a", "id": 1}] 2024-03-25T19:40:22.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.786+0000 7ff1cc295640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:22.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.786+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d209ed26-5603-4a13-a52f-7a40169d3c6a", "id": 1} v 0) v1 -- 0x7ff1c40aab00 con 0x7ff1c4142a20 2024-03-25T19:40:22.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.841+0000 7f4bc9813040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:22.900 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:22.900+0000 7f4bc9813040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:22.941 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.940+0000 7ff1baffd640 1 -- 172.21.15.178:0/1820815899 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d209ed26-5603-4a13-a52f-7a40169d3c6a", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3249867699 0 940483276) 0x7ff1bc005bd0 con 0x7ff1c4142a20 2024-03-25T19:40:22.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.940+0000 7ff1baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d209ed26-5603-4a13-a52f-7a40169d3c6a", "id": 1}] 2024-03-25T19:40:22.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.940+0000 7ff1baffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:22.942 INFO:teuthology.orchestra.run.smithi178.stdout:1 2024-03-25T19:40:22.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 10 librados: watch_flush enter 2024-03-25T19:40:22.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 10 librados: watch_flush exit 2024-03-25T19:40:22.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T19:40:22.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 10 client.4126.objecter successfully canceled tick 2024-03-25T19:40:22.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 10 monclient: shutdown 2024-03-25T19:40:22.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:22.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 >> v1:172.21.15.178:6789/0 conn(0x7ff1c4142a20 legacy=0x7ff1c413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 shutdown_connections 2024-03-25T19:40:22.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.941+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 >> 172.21.15.178:0/1820815899 conn(0x7ff1c40668e0 msgr2=0x7ff1c40b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:22.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.942+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 shutdown_connections 2024-03-25T19:40:22.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.942+0000 7ff1cc295640 1 -- 172.21.15.178:0/1820815899 wait complete. 2024-03-25T19:40:22.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:22.942+0000 7ff1cc295640 1 librados: shutdown 2024-03-25T19:40:22.944 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:22.942+0000 7fb51062b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:40:22.952 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 39e9bb9d-3b9c-47bb-a410-56019ae94e67 2 2024-03-25T19:40:23.084 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:23.084+0000 7fb51062b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:40:23.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.093+0000 7fcefd674640 10 monclient: get_monmap_and_config 2024-03-25T19:40:23.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.093+0000 7fcefd674640 10 monclient: build_initial_monmap 2024-03-25T19:40:23.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.093+0000 7fcefd674640 10 monclient: monmap: 2024-03-25T19:40:23.093 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:23.093 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.094 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:40:23.095109+0000 2024-03-25T19:40:23.094 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:40:23.095109+0000 2024-03-25T19:40:23.094 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:23.094 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.094 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:23.094 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:23.094 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:23.094 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.093+0000 7fcefd674640 1 Processor -- start 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.093+0000 7fcefd674640 1 -- start start 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.093+0000 7fcefd674640 10 monclient: init 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 10 monclient(hunting): picked mon.noname-c con 0x7fcef80a9920 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 10 monclient(hunting): picked mon.noname-b con 0x7fcef80b45a0 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 10 monclient(hunting): picked mon.noname-a con 0x7fcef8146450 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef806b100 con 0x7fcef8146450 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef80b2940 con 0x7fcef80a9920 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef813fec0 con 0x7fcef80b45a0 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.094+0000 7fcefd674640 10 monclient(hunting): authenticate will time out at 4178.245064s 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.095+0000 7fcef67fc640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fcef80b45a0 0x7fcef8145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.178:37464/0 (socket says 172.21.15.178:37464) 2024-03-25T19:40:23.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.095+0000 7fcef67fc640 1 -- 172.21.15.178:0/1805031296 learned_addr learned my addr 172.21.15.178:0/1805031296 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:23.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178321694 0 0) 0x7fcef813fec0 con 0x7fcef80b45a0 2024-03-25T19:40:23.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee0003520 con 0x7fcef80b45a0 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208080393 0 0) 0x7fcef806b100 con 0x7fcef8146450 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.097+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef813fec0 con 0x7fcef8146450 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.098+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504916963 0 0) 0x7fcef80b2940 con 0x7fcef80a9920 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.098+0000 7fcef5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.098+0000 7fcef5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.098+0000 7fcef5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.098+0000 7fcef5ffb640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:40:23.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.098+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef806b100 con 0x7fcef80a9920 2024-03-25T19:40:23.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2646587266 0 0) 0x7fcee0003520 con 0x7fcef80b45a0 2024-03-25T19:40:23.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 >> v1:172.21.15.178:6789/0 conn(0x7fcef80a9920 legacy=0x7fcef813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 >> v1:172.21.15.110:6789/0 conn(0x7fcef8146450 legacy=0x7fcef814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 1 monclient: found mon.noname-b 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef8140050 con 0x7fcef80b45a0 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcee8001690 con 0x7fcef80b45a0 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcefd674640 5 monclient: authenticate success, global_id 4130 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcefd674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcef5ffb640 10 monclient: dump: 2024-03-25T19:40:23.102 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.101+0000 7fcefd674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.127+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcee80017f0 con 0x7fcef80b45a0 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.127+0000 7fcef5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.127+0000 7fcef5ffb640 1 -- 172.21.15.178:0/1805031296 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcee8001a60 con 0x7fcef80b45a0 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.127+0000 7fcef5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.127+0000 7fcef5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.127+0000 7fcef5ffb640 10 monclient: dump: 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.128 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.127+0000 7fcefd674640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:23.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.127+0000 7fcefd674640 4 set_mon_vals no callback set 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.128+0000 7fcefd674640 10 monclient: shutdown 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.128+0000 7fcefd674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.128+0000 7fcefd674640 1 -- 172.21.15.178:0/1805031296 >> v1:172.21.15.110:6790/0 conn(0x7fcef80b45a0 legacy=0x7fcef8145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.128+0000 7fcefd674640 1 -- 172.21.15.178:0/1805031296 shutdown_connections 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.128+0000 7fcefd674640 1 -- 172.21.15.178:0/1805031296 >> 172.21.15.178:0/1805031296 conn(0x7fcef80b7f40 msgr2=0x7fcef80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.128+0000 7fcefd674640 1 -- 172.21.15.178:0/1805031296 shutdown_connections 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.128+0000 7fcefd674640 1 -- 172.21.15.178:0/1805031296 wait complete. 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.129+0000 7fcefd674640 10 monclient: build_initial_monmap 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.129+0000 7fcefd674640 10 monclient: monmap: 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:23.132 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.129+0000 7fcefd674640 1 librados: starting msgr at 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.129+0000 7fcefd674640 1 librados: starting objecter 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.129+0000 7fcefd674640 1 Processor -- start 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 1 -- start start 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 1 librados: setting wanted keys 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 1 librados: calling monclient init 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 10 monclient: init 2024-03-25T19:40:23.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 10 monclient(hunting): picked mon.noname-a con 0x7fcef80a9920 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 10 monclient(hunting): picked mon.noname-b con 0x7fcef80b45a0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 10 monclient(hunting): picked mon.noname-c con 0x7fcef8146450 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef80ac860 con 0x7fcef80a9920 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef81d7330 con 0x7fcef80b45a0 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef8141790 con 0x7fcef8146450 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.130+0000 7fcefd674640 10 monclient(hunting): authenticate will time out at 4178.281179s 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcef67fc640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fcef80b45a0 0x7fcef81d77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34272/0 (socket says 172.21.15.178:34272) 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcef77fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fcef8146450 0x7fcef8137380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.178:37480/0 (socket says 172.21.15.178:37480) 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcef67fc640 1 -- 172.21.15.178:0/971879070 learned_addr learned my addr 172.21.15.178:0/971879070 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcedf7fe640 10 client.?.objecter ms_handle_connect 0x7fcef8146450 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcedf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.134 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcedf7fe640 10 client.?.objecter ms_handle_connect 0x7fcef80b45a0 2024-03-25T19:40:23.135 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcedf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.135 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcedf7fe640 10 client.?.objecter ms_handle_connect 0x7fcef80a9920 2024-03-25T19:40:23.135 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.131+0000 7fcedf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.133+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554651522 0 0) 0x7fcef8141790 con 0x7fcef8146450 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.133+0000 7fcedf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.133+0000 7fcedf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.133+0000 7fcedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.133+0000 7fcedf7fe640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.133+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee4004eb0 con 0x7fcef8146450 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727799558 0 0) 0x7fcef81d7330 con 0x7fcef80b45a0 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:40:23.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef8141790 con 0x7fcef80b45a0 2024-03-25T19:40:23.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338585964 0 0) 0x7fcef80ac860 con 0x7fcef80a9920 2024-03-25T19:40:23.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:40:23.137 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.134+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef81d7330 con 0x7fcef80a9920 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.138+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2298595865 0 0) 0x7fcee4004eb0 con 0x7fcef8146450 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.138+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcef80ac860 con 0x7fcef8146450 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.138+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcef0001bd0 con 0x7fcef8146450 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.138+0000 7fcedf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.138+0000 7fcedf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.138+0000 7fcedf7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.140 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.138+0000 7fcedf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3731209714 0 0) 0x7fcef8141790 con 0x7fcef80b45a0 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcee4004eb0 con 0x7fcef80b45a0 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcee8001b00 con 0x7fcef80b45a0 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.141 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2192322013 0 0) 0x7fcef81d7330 con 0x7fcef80a9920 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcef8141790 con 0x7fcef80a9920 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fceec001830 con 0x7fcef80a9920 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.139+0000 7fcedf7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.142 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.143 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.143 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.143 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.143 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.143 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.143 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.143 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2116675679 0 0) 0x7fcef80ac860 con 0x7fcef8146450 2024-03-25T19:40:23.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 >> v1:172.21.15.178:6789/0 conn(0x7fcef80b45a0 legacy=0x7fcef81d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 >> v1:172.21.15.110:6789/0 conn(0x7fcef80a9920 legacy=0x7fcef8144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 1 monclient: found mon.c 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef8141920 con 0x7fcef8146450 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcedf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 5 monclient: authenticate success, global_id 4133 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 10 monclient: _renew_subs 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef8145540 con 0x7fcef8146450 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 10 monclient: _renew_subs 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:23.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef81d80d0 con 0x7fcef8146450 2024-03-25T19:40:23.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.141+0000 7fcefd674640 1 librados: init done 2024-03-25T19:40:23.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.142+0000 7fcefd674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.142+0000 7fcefd674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.142+0000 7fcefd674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:23.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.142+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef80ab430 con 0x7fcef8146450 2024-03-25T19:40:23.202 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:23.202+0000 7fb51062b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:40:23.248 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:23.248+0000 7fb51062b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:40:23.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:23.298+0000 7f4bc9813040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.331+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef00020c0 con 0x7fcef8146450 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.331+0000 7fcedf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.331+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcef0003970 con 0x7fcef8146450 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.331+0000 7fcedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.331+0000 7fcedf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.331+0000 7fcedf7fe640 10 monclient: dump: 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.332 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.333 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.333 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.333 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.333 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.333 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.331+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcef0003f20 con 0x7fcef8146450 2024-03-25T19:40:23.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.331+0000 7fcef4ff9640 4 set_mon_vals no callback set 2024-03-25T19:40:23.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.336+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3414696916 0 0) 0x7fcef0006770 con 0x7fcef8146450 2024-03-25T19:40:23.336 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.336+0000 7fcedf7fe640 10 client.4133.objecter ms_dispatch 0x7fcef8000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:40:23.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.336+0000 7fcedf7fe640 3 client.4133.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:40:23.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.336+0000 7fcedf7fe640 3 client.4133.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:40:23.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.336+0000 7fcedf7fe640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T19:40:23.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.336+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcef0005b40 con 0x7fcef8146450 2024-03-25T19:40:23.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.336+0000 7fcedf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.336+0000 7fcedf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:23.359 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:23.359+0000 7fb51062b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:23.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.408+0000 7fcefd674640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "39e9bb9d-3b9c-47bb-a410-56019ae94e67", "id": 2}] 2024-03-25T19:40:23.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.408+0000 7fcefd674640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "39e9bb9d-3b9c-47bb-a410-56019ae94e67", "id": 2}] 2024-03-25T19:40:23.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.408+0000 7fcefd674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:23.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.408+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd new", "uuid": "39e9bb9d-3b9c-47bb-a410-56019ae94e67", "id": 2} v 0) v1 -- 0x7fcef804f730 con 0x7fcef8146450 2024-03-25T19:40:23.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:23.434+0000 7f4bc9813040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:40:23.456 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:23.456+0000 7fb51062b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:40:23.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.540+0000 7fcedf7fe640 1 -- 172.21.15.178:0/971879070 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "39e9bb9d-3b9c-47bb-a410-56019ae94e67", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3497382961 0 216599125) 0x7fcef0006350 con 0x7fcef8146450 2024-03-25T19:40:23.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.540+0000 7fcedf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "39e9bb9d-3b9c-47bb-a410-56019ae94e67", "id": 2}] 2024-03-25T19:40:23.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.540+0000 7fcedf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:23.541 INFO:teuthology.orchestra.run.smithi178.stdout:2 2024-03-25T19:40:23.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 10 librados: watch_flush enter 2024-03-25T19:40:23.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 10 librados: watch_flush exit 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 10 client.4133.objecter successfully canceled tick 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 10 monclient: shutdown 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 >> v1:172.21.15.110:6790/0 conn(0x7fcef8146450 legacy=0x7fcef8137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 shutdown_connections 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 >> 172.21.15.178:0/971879070 conn(0x7fcef80b7f40 msgr2=0x7fcef80b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 shutdown_connections 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 1 -- 172.21.15.178:0/971879070 wait complete. 2024-03-25T19:40:23.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.541+0000 7fcefd674640 1 librados: shutdown 2024-03-25T19:40:23.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 9ebeaa83-4e6a-4770-9656-4f0d65dd06fc 3 2024-03-25T19:40:23.561 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:23.562+0000 7f4bc9813040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:40:23.607 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:23.608+0000 7f4bc9813040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:40:23.616 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:23.616+0000 7fb51062b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.651+0000 7f221b8c1640 10 monclient: get_monmap_and_config 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.651+0000 7f221b8c1640 10 monclient: build_initial_monmap 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.651+0000 7f221b8c1640 10 monclient: monmap: 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:40:23.653614+0000 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:40:23.653614+0000 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.652 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:23.653 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:23.653 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:23.653 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.652+0000 7f221b8c1640 1 Processor -- start 2024-03-25T19:40:23.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.652+0000 7f221b8c1640 1 -- start start 2024-03-25T19:40:23.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.652+0000 7f221b8c1640 10 monclient: init 2024-03-25T19:40:23.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:23.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:23.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f22140b4700 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:23.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f2214142a20 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:23.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f22140a96b0 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:23.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22140a9d60 con 0x7f2214142a20 2024-03-25T19:40:23.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221413ff10 con 0x7f22140b4700 2024-03-25T19:40:23.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22140aa1f0 con 0x7f22140a96b0 2024-03-25T19:40:23.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:23.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f221b8c1640 10 monclient(hunting): authenticate will time out at 4178.803766s 2024-03-25T19:40:23.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f2219e5e640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f22140a96b0 0x7f221413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.178:37492/0 (socket says 172.21.15.178:37492) 2024-03-25T19:40:23.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.653+0000 7f2219e5e640 1 -- 172.21.15.178:0/3108623140 learned_addr learned my addr 172.21.15.178:0/3108623140 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154856451 0 0) 0x7f22140aa1f0 con 0x7f22140a96b0 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2200003520 con 0x7f22140a96b0 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981186452 0 0) 0x7f221413ff10 con 0x7f22140b4700 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:40:23.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22140aa1f0 con 0x7f22140b4700 2024-03-25T19:40:23.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910119409 0 0) 0x7f22140a9d60 con 0x7f2214142a20 2024-03-25T19:40:23.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:40:23.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.656+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f221413ff10 con 0x7f2214142a20 2024-03-25T19:40:23.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.659+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1275136587 0 0) 0x7f22140aa1f0 con 0x7f22140b4700 2024-03-25T19:40:23.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.659+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 >> v1:172.21.15.110:6790/0 conn(0x7f22140a96b0 legacy=0x7f221413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.659+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 >> v1:172.21.15.110:6789/0 conn(0x7f2214142a20 legacy=0x7f221413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.659+0000 7f220bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.659+0000 7f220bfff640 1 monclient: found mon.noname-c 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22140b4ae0 con 0x7f22140b4700 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 10 monclient: _finish_auth 0 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f221b8c1640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2204001670 con 0x7f22140b4700 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f220bfff640 10 monclient: dump: 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.661 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.662 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.662 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.662 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.662 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.662 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.662 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.662 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.660+0000 7f221b8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:23.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.661+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22040017d0 con 0x7f22140b4700 2024-03-25T19:40:23.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.661+0000 7f220bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:23.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.661+0000 7f220bfff640 1 -- 172.21.15.178:0/3108623140 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2204001be0 con 0x7f22140b4700 2024-03-25T19:40:23.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.661+0000 7f220bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.661+0000 7f220bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.661+0000 7f220bfff640 10 monclient: dump: 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.661+0000 7f221b8c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.661+0000 7f221b8c1640 4 set_mon_vals no callback set 2024-03-25T19:40:23.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.662+0000 7f221b8c1640 10 monclient: shutdown 2024-03-25T19:40:23.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.662+0000 7f221b8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:23.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.662+0000 7f221b8c1640 1 -- 172.21.15.178:0/3108623140 >> v1:172.21.15.178:6789/0 conn(0x7f22140b4700 legacy=0x7f2214142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.662+0000 7f221b8c1640 1 -- 172.21.15.178:0/3108623140 shutdown_connections 2024-03-25T19:40:23.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.662+0000 7f221b8c1640 1 -- 172.21.15.178:0/3108623140 >> 172.21.15.178:0/3108623140 conn(0x7f22140668e0 msgr2=0x7f2214066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:23.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.662+0000 7f221b8c1640 1 -- 172.21.15.178:0/3108623140 shutdown_connections 2024-03-25T19:40:23.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.662+0000 7f221b8c1640 1 -- 172.21.15.178:0/3108623140 wait complete. 2024-03-25T19:40:23.665 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:23.663+0000 7fb51062b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 10 monclient: build_initial_monmap 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 10 monclient: monmap: 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:23.668 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 1 librados: starting msgr at 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 1 librados: starting objecter 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 1 Processor -- start 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 1 -- start start 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 1 librados: setting wanted keys 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 1 librados: calling monclient init 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.663+0000 7f221b8c1640 10 monclient: init 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f22140a96b0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f22140b4700 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:23.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f2214142a20 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22140ac120 con 0x7f22140b4700 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22140ac400 con 0x7f22140a96b0 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2214136330 con 0x7f2214142a20 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f221b8c1640 10 monclient(hunting): authenticate will time out at 4178.814849s 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f2219e5e640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f2214142a20 0x7f2214135860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.178:37508/0 (socket says 172.21.15.178:37508) 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.664+0000 7f2219e5e640 1 -- 172.21.15.178:0/2133307310 learned_addr learned my addr 172.21.15.178:0/2133307310 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.665+0000 7f2209ffb640 10 client.?.objecter ms_handle_connect 0x7f2214142a20 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.665+0000 7f2209ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.665+0000 7f2209ffb640 10 client.?.objecter ms_handle_connect 0x7f22140b4700 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.665+0000 7f2209ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.665+0000 7f2209ffb640 10 client.?.objecter ms_handle_connect 0x7f22140a96b0 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.665+0000 7f2209ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726351656 0 0) 0x7f2214136330 con 0x7f2214142a20 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f8004eb0 con 0x7f2214142a20 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430529202 0 0) 0x7f22140ac120 con 0x7f22140b4700 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2214136330 con 0x7f22140b4700 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881552517 0 0) 0x7f22140ac400 con 0x7f22140a96b0 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:40:23.672 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.667+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22140ac120 con 0x7f22140a96b0 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.671+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3520842760 0 0) 0x7f21f8004eb0 con 0x7f2214142a20 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.671+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22140ac400 con 0x7f2214142a20 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.671+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f220c001bb0 con 0x7f2214142a20 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.673 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3390512322 0 0) 0x7f2214136330 con 0x7f22140b4700 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21f8004eb0 con 0x7f22140b4700 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2210001870 con 0x7f22140b4700 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.674 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1398695643 0 0) 0x7f22140ac120 con 0x7f22140a96b0 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2214136330 con 0x7f22140a96b0 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2204001ae0 con 0x7f22140a96b0 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.672+0000 7f2209ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.675 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.676 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.676 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.676 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.676 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.676 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.676 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.676 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.676 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3474638050 0 0) 0x7f2214136330 con 0x7f22140a96b0 2024-03-25T19:40:23.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 >> v1:172.21.15.110:6790/0 conn(0x7f2214142a20 legacy=0x7f2214135860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 >> v1:172.21.15.110:6789/0 conn(0x7f22140b4700 legacy=0x7f22140ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:23.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 1 monclient: found mon.b 2024-03-25T19:40:23.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:23.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2214135030 con 0x7f22140a96b0 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f2209ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 5 monclient: authenticate success, global_id 4138 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 10 monclient: _renew_subs 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2214135f70 con 0x7f22140a96b0 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 10 monclient: _renew_subs 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22141e3270 con 0x7f22140a96b0 2024-03-25T19:40:23.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.675+0000 7f221b8c1640 1 librados: init done 2024-03-25T19:40:23.679 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.676+0000 7f221b8c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.679 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.676+0000 7f221b8c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.679 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.676+0000 7f221b8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:23.679 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.676+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221404f730 con 0x7f22140a96b0 2024-03-25T19:40:23.725 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:23.726+0000 7f4bc9813040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:23.776 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2204001210 con 0x7f22140a96b0 2024-03-25T19:40:23.776 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:23.776 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2204003a40 con 0x7f22140a96b0 2024-03-25T19:40:23.776 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.776 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:23.776 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 10 monclient: dump: 2024-03-25T19:40:23.776 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2204003f30 con 0x7f22140a96b0 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f220b7fe640 4 set_mon_vals no callback set 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3046245071 0 0) 0x7f2204005520 con 0x7f22140a96b0 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 10 client.4138.objecter ms_dispatch 0x7f2214000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:40:23.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:40:23.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T19:40:23.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f22040058c0 con 0x7f22140a96b0 2024-03-25T19:40:23.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.775+0000 7f2209ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:23.821 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:23.821+0000 7f4bc9813040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:40:23.826 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:23.825+0000 7fb51062b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:40:23.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.848+0000 7f221b8c1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9ebeaa83-4e6a-4770-9656-4f0d65dd06fc", "id": 3}] 2024-03-25T19:40:23.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.848+0000 7f221b8c1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9ebeaa83-4e6a-4770-9656-4f0d65dd06fc", "id": 3}] 2024-03-25T19:40:23.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.848+0000 7f221b8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:23.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.848+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9ebeaa83-4e6a-4770-9656-4f0d65dd06fc", "id": 3} v 0) v1 -- 0x7f2214136160 con 0x7f22140a96b0 2024-03-25T19:40:23.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.934+0000 7f2209ffb640 1 -- 172.21.15.178:0/2133307310 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9ebeaa83-4e6a-4770-9656-4f0d65dd06fc", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1520799075 0 525046306) 0x7f2204005160 con 0x7f22140a96b0 2024-03-25T19:40:23.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.934+0000 7f2209ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9ebeaa83-4e6a-4770-9656-4f0d65dd06fc", "id": 3}] 2024-03-25T19:40:23.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.934+0000 7f2209ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:23.935 INFO:teuthology.orchestra.run.smithi178.stdout:3 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.935+0000 7f221b8c1640 10 librados: watch_flush enter 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.935+0000 7f221b8c1640 10 librados: watch_flush exit 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.935+0000 7f221b8c1640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.935+0000 7f221b8c1640 10 client.4138.objecter successfully canceled tick 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.936+0000 7f221b8c1640 10 monclient: shutdown 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.936+0000 7f221b8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.936+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 >> v1:172.21.15.178:6789/0 conn(0x7f22140a96b0 legacy=0x7f2214134180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.936+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 shutdown_connections 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.936+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 >> 172.21.15.178:0/2133307310 conn(0x7f22140668e0 msgr2=0x7f22140b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.936+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 shutdown_connections 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.936+0000 7f221b8c1640 1 -- 172.21.15.178:0/2133307310 wait complete. 2024-03-25T19:40:23.937 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:23.936+0000 7f221b8c1640 1 librados: shutdown 2024-03-25T19:40:23.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new f422fd1b-9010-47e4-8509-0ad530571a80 4 2024-03-25T19:40:23.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:23.984+0000 7f4bc9813040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:24.028 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:24.028+0000 7f4bc9813040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:24.049 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.049+0000 7f081c523640 10 monclient: get_monmap_and_config 2024-03-25T19:40:24.049 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.049+0000 7f081c523640 10 monclient: build_initial_monmap 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.049+0000 7f081c523640 10 monclient: monmap: 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:40:24.051282+0000 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:40:24.051282+0000 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:24.050 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 1 Processor -- start 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 1 -- start start 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 10 monclient: init 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 10 monclient(hunting): picked mon.noname-b con 0x7f08140b4700 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 10 monclient(hunting): picked mon.noname-a con 0x7f0814142a20 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 10 monclient(hunting): picked mon.noname-c con 0x7f08140a96b0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140a9d60 con 0x7f0814142a20 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081413ff10 con 0x7f08140a96b0 2024-03-25T19:40:24.052 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140aa1f0 con 0x7f08140b4700 2024-03-25T19:40:24.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:24.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.050+0000 7f081c523640 10 monclient(hunting): authenticate will time out at 4179.201459s 2024-03-25T19:40:24.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.051+0000 7f081a2bf640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f08140b4700 0x7f0814142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.178:37512/0 (socket says 172.21.15.178:37512) 2024-03-25T19:40:24.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.051+0000 7f081a2bf640 1 -- 172.21.15.178:0/1932816845 learned_addr learned my addr 172.21.15.178:0/1932816845 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:24.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.053+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43405389 0 0) 0x7f08140aa1f0 con 0x7f08140b4700 2024-03-25T19:40:24.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.053+0000 7f08192bd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.053+0000 7f08192bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.053+0000 7f08192bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.053+0000 7f08192bd640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.053+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0800003520 con 0x7f08140b4700 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828791796 0 0) 0x7f08140a9d60 con 0x7f0814142a20 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08140aa1f0 con 0x7f0814142a20 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785907409 0 0) 0x7f081413ff10 con 0x7f08140a96b0 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:40:24.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.054+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08140a9d60 con 0x7f08140a96b0 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180266485 0 0) 0x7f0800003520 con 0x7f08140b4700 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 >> v1:172.21.15.178:6789/0 conn(0x7f08140a96b0 legacy=0x7f081413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 >> v1:172.21.15.110:6789/0 conn(0x7f0814142a20 legacy=0x7f081413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 1 monclient: found mon.noname-b 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140b4ae0 con 0x7f08140b4700 2024-03-25T19:40:24.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 10 monclient: _finish_auth 0 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0810001a10 con 0x7f08140b4700 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f081c523640 5 monclient: authenticate success, global_id 4142 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f08192bd640 10 monclient: dump: 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.059 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.060 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.060 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.057+0000 7f081c523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0810001b70 con 0x7f08140b4700 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f08192bd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f08192bd640 1 -- 172.21.15.178:0/1932816845 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0810001f80 con 0x7f08140b4700 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f08192bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f08192bd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f08192bd640 10 monclient: dump: 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 4 set_mon_vals no callback set 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 10 monclient: shutdown 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 1 -- 172.21.15.178:0/1932816845 >> v1:172.21.15.110:6790/0 conn(0x7f08140b4700 legacy=0x7f0814142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 1 -- 172.21.15.178:0/1932816845 shutdown_connections 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 1 -- 172.21.15.178:0/1932816845 >> 172.21.15.178:0/1932816845 conn(0x7f08140668e0 msgr2=0x7f0814066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 1 -- 172.21.15.178:0/1932816845 shutdown_connections 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.059+0000 7f081c523640 1 -- 172.21.15.178:0/1932816845 wait complete. 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 10 monclient: build_initial_monmap 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 10 monclient: monmap: 2024-03-25T19:40:24.064 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 1 librados: starting msgr at 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 1 librados: starting objecter 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 1 Processor -- start 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 1 -- start start 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 1 librados: setting wanted keys 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 1 librados: calling monclient init 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.060+0000 7f081c523640 10 monclient: init 2024-03-25T19:40:24.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 10 monclient(hunting): picked mon.noname-c con 0x7f08140a96b0 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 10 monclient(hunting): picked mon.noname-b con 0x7f08140b4700 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 10 monclient(hunting): picked mon.noname-a con 0x7f0814142a20 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140addd0 con 0x7f0814142a20 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140ae040 con 0x7f08140b4700 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140ae320 con 0x7f08140a96b0 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.061+0000 7f081c523640 10 monclient(hunting): authenticate will time out at 4179.212061s 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.062+0000 7f081a2bf640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f08140a96b0 0x7f08140ac860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.178:37516/0 (socket says 172.21.15.178:37516) 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.062+0000 7f081a2bf640 1 -- 172.21.15.178:0/3384719347 learned_addr learned my addr 172.21.15.178:0/3384719347 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.062+0000 7f080affd640 10 client.?.objecter ms_handle_connect 0x7f08140a96b0 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.062+0000 7f080affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.062+0000 7f080affd640 10 client.?.objecter ms_handle_connect 0x7f08140b4700 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.062+0000 7f080affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.062+0000 7f080affd640 10 client.?.objecter ms_handle_connect 0x7f0814142a20 2024-03-25T19:40:24.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.062+0000 7f080affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188582113 0 0) 0x7f08140ae040 con 0x7f08140b4700 2024-03-25T19:40:24.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f8004eb0 con 0x7f08140b4700 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373961622 0 0) 0x7f08140ae320 con 0x7f08140a96b0 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08140ae040 con 0x7f08140a96b0 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551824627 0 0) 0x7f08140addd0 con 0x7f0814142a20 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:40:24.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.064+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08140ae320 con 0x7f0814142a20 2024-03-25T19:40:24.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3624255310 0 0) 0x7f07f8004eb0 con 0x7f08140b4700 2024-03-25T19:40:24.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08140addd0 con 0x7f08140b4700 2024-03-25T19:40:24.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0804001970 con 0x7f08140b4700 2024-03-25T19:40:24.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 10 monclient(hunting): dump: 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 398426652 0 0) 0x7f08140ae040 con 0x7f08140a96b0 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07f8004eb0 con 0x7f08140a96b0 2024-03-25T19:40:24.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0810001ec0 con 0x7f08140a96b0 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.069+0000 7f080affd640 10 monclient(hunting): dump: 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.070+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 6881037 0 0) 0x7f08140ae320 con 0x7f0814142a20 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.070+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08140ae040 con 0x7f0814142a20 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.070+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f080c001850 con 0x7f0814142a20 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.070+0000 7f080affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.070+0000 7f080affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.070+0000 7f080affd640 10 monclient(hunting): dump: 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.073 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1008878416 0 0) 0x7f07f8004eb0 con 0x7f08140a96b0 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 >> v1:172.21.15.178:6789/0 conn(0x7f08140b4700 legacy=0x7f081413a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 >> v1:172.21.15.110:6789/0 conn(0x7f0814142a20 legacy=0x7f08140ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 1 monclient: found mon.c 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140ad010 con 0x7f08140a96b0 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 10 monclient: _finish_auth 0 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f080affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 5 monclient: authenticate success, global_id 4145 2024-03-25T19:40:24.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 10 monclient: _renew_subs 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08140abcc0 con 0x7f08140a96b0 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 10 monclient: _renew_subs 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08141e3270 con 0x7f08140a96b0 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.073+0000 7f081c523640 1 librados: init done 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.074+0000 7f081c523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.074+0000 7f081c523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.074+0000 7f081c523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.074+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08140b2680 con 0x7f08140a96b0 2024-03-25T19:40:24.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0810001570 con 0x7f08140a96b0 2024-03-25T19:40:24.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:24.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0810004fb0 con 0x7f08140a96b0 2024-03-25T19:40:24.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:24.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 10 monclient: dump: 2024-03-25T19:40:24.077 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0810005520 con 0x7f08140a96b0 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f0818abc640 4 set_mon_vals no callback set 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3095928526 0 0) 0x7f0810006c50 con 0x7f08140a96b0 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 10 client.4145.objecter ms_dispatch 0x7f0814000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:40:24.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 3 client.4145.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:40:24.079 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 3 client.4145.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:40:24.079 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.077+0000 7f080affd640 20 client.4145.objecter dump_active .. 0 homeless 2024-03-25T19:40:24.168 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.168+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0810006e60 con 0x7f08140a96b0 2024-03-25T19:40:24.168 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.168+0000 7f080affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.169 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.168+0000 7f080affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:24.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:24.188+0000 7f4bc9813040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:40:24.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.241+0000 7f081c523640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f422fd1b-9010-47e4-8509-0ad530571a80", "id": 4}] 2024-03-25T19:40:24.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.241+0000 7f081c523640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f422fd1b-9010-47e4-8509-0ad530571a80", "id": 4}] 2024-03-25T19:40:24.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.241+0000 7f081c523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.241+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f422fd1b-9010-47e4-8509-0ad530571a80", "id": 4} v 0) v1 -- 0x7f081404f730 con 0x7f08140a96b0 2024-03-25T19:40:24.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.316+0000 7f080affd640 1 -- 172.21.15.178:0/3384719347 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f422fd1b-9010-47e4-8509-0ad530571a80", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1479318387 0 1697012583) 0x7f0810006920 con 0x7f08140a96b0 2024-03-25T19:40:24.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.316+0000 7f080affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f422fd1b-9010-47e4-8509-0ad530571a80", "id": 4}] 2024-03-25T19:40:24.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.316+0000 7f080affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:24.317 INFO:teuthology.orchestra.run.smithi178.stdout:4 2024-03-25T19:40:24.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 10 librados: watch_flush enter 2024-03-25T19:40:24.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 10 librados: watch_flush exit 2024-03-25T19:40:24.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 20 client.4145.objecter shutdown clearing up homeless session... 2024-03-25T19:40:24.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 10 client.4145.objecter successfully canceled tick 2024-03-25T19:40:24.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 10 monclient: shutdown 2024-03-25T19:40:24.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:24.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 >> v1:172.21.15.110:6790/0 conn(0x7f08140a96b0 legacy=0x7f08140ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 shutdown_connections 2024-03-25T19:40:24.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 >> 172.21.15.178:0/3384719347 conn(0x7f08140668e0 msgr2=0x7f08140b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:24.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 shutdown_connections 2024-03-25T19:40:24.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 1 -- 172.21.15.178:0/3384719347 wait complete. 2024-03-25T19:40:24.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.317+0000 7f081c523640 1 librados: shutdown 2024-03-25T19:40:24.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new c08330a9-07e8-49ab-b0c7-fbade08e842d 5 2024-03-25T19:40:24.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.441+0000 7f0519e63640 10 monclient: get_monmap_and_config 2024-03-25T19:40:24.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.441+0000 7f0519e63640 10 monclient: build_initial_monmap 2024-03-25T19:40:24.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.441+0000 7f0519e63640 10 monclient: monmap: 2024-03-25T19:40:24.442 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:24.442 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.442 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:40:24.443653+0000 2024-03-25T19:40:24.442 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:40:24.443653+0000 2024-03-25T19:40:24.443 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:24.443 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.443 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:24.443 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:24.443 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:24.443 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.442+0000 7f0519e63640 1 Processor -- start 2024-03-25T19:40:24.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.442+0000 7f0519e63640 1 -- start start 2024-03-25T19:40:24.444 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.442+0000 7f0519e63640 10 monclient: init 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 10 monclient(hunting): picked mon.noname-b con 0x7f05140b4e90 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 10 monclient(hunting): picked mon.noname-a con 0x7f05140b4340 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 10 monclient(hunting): picked mon.noname-c con 0x7f051413fd00 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140b5270 con 0x7f05140b4340 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140ab330 con 0x7f051413fd00 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140a9d80 con 0x7f05140b4e90 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.443+0000 7f0519e63640 10 monclient(hunting): authenticate will time out at 4179.594307s 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.444+0000 7f0513fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f051413fd00 0x7f0514146150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34316/0 (socket says 172.21.15.178:34316) 2024-03-25T19:40:24.445 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.444+0000 7f0513fff640 1 -- 172.21.15.178:0/908055421 learned_addr learned my addr 172.21.15.178:0/908055421 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:24.448 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.448+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870494391 0 0) 0x7f05140a9d80 con 0x7f05140b4e90 2024-03-25T19:40:24.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.448+0000 7f05127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.448+0000 7f05127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.448+0000 7f05127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.448+0000 7f05127fc640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:40:24.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.448+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc003520 con 0x7f05140b4e90 2024-03-25T19:40:24.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.449+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509579310 0 0) 0x7f05140ab330 con 0x7f051413fd00 2024-03-25T19:40:24.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.449+0000 7f05127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.449+0000 7f05127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.449+0000 7f05127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.449+0000 7f05127fc640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:40:24.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.449+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05140a9d80 con 0x7f051413fd00 2024-03-25T19:40:24.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.450+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330012291 0 0) 0x7f05140b5270 con 0x7f05140b4340 2024-03-25T19:40:24.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.450+0000 7f05127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.450+0000 7f05127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.450+0000 7f05127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.450+0000 7f05127fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:40:24.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.450+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05140ab330 con 0x7f05140b4340 2024-03-25T19:40:24.452 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4250808209 0 0) 0x7f04fc003520 con 0x7f05140b4e90 2024-03-25T19:40:24.452 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 >> v1:172.21.15.178:6789/0 conn(0x7f051413fd00 legacy=0x7f0514146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 >> v1:172.21.15.110:6789/0 conn(0x7f05140b4340 legacy=0x7f0514145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 1 monclient: found mon.noname-b 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140a96b0 con 0x7f05140b4e90 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0508001690 con 0x7f05140b4e90 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f0519e63640 5 monclient: authenticate success, global_id 4148 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f0519e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:24.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f05127fc640 10 monclient: dump: 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.452+0000 7f0519e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:24.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05080017f0 con 0x7f05140b4e90 2024-03-25T19:40:24.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f05127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:24.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f05127fc640 1 -- 172.21.15.178:0/908055421 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0508001c00 con 0x7f05140b4e90 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 4 set_mon_vals no callback set 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f05127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f05127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f05127fc640 10 monclient: dump: 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.457 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 10 monclient: shutdown 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 1 -- 172.21.15.178:0/908055421 >> v1:172.21.15.110:6790/0 conn(0x7f05140b4e90 legacy=0x7f051413f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 1 -- 172.21.15.178:0/908055421 shutdown_connections 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 1 -- 172.21.15.178:0/908055421 >> 172.21.15.178:0/908055421 conn(0x7f0514066ac0 msgr2=0x7f0514066ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 1 -- 172.21.15.178:0/908055421 shutdown_connections 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.454+0000 7f0519e63640 1 -- 172.21.15.178:0/908055421 wait complete. 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.455+0000 7f0519e63640 10 monclient: build_initial_monmap 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.455+0000 7f0519e63640 10 monclient: monmap: 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-03-25T19:40:24.458 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.455+0000 7f0519e63640 1 librados: starting msgr at 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.455+0000 7f0519e63640 1 librados: starting objecter 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.455+0000 7f0519e63640 1 Processor -- start 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 1 -- start start 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 1 librados: setting wanted keys 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 1 librados: calling monclient init 2024-03-25T19:40:24.459 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 10 monclient: init 2024-03-25T19:40:24.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:24.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 10 monclient(hunting): picked mon.noname-b con 0x7f05140b4e90 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 10 monclient(hunting): picked mon.noname-a con 0x7f051413fd00 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 10 monclient(hunting): picked mon.noname-c con 0x7f0514134020 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0514135380 con 0x7f051413fd00 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.456+0000 7f0519e63640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05141355f0 con 0x7f05140b4e90 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f0519e63640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05141358d0 con 0x7f0514134020 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f0519e63640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f0519e63640 10 monclient(hunting): authenticate will time out at 4179.607543s 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f05137fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f05140b4e90 0x7f0514141350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34326/0 (socket says 172.21.15.178:34326) 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f05137fe640 1 -- 172.21.15.178:0/576404234 learned_addr learned my addr 172.21.15.178:0/576404234 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f04fbfff640 10 client.?.objecter ms_handle_connect 0x7f05140b4e90 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f04fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f04fbfff640 10 client.?.objecter ms_handle_connect 0x7f0514134020 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.457+0000 7f04fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.458+0000 7f04fbfff640 10 client.?.objecter ms_handle_connect 0x7f051413fd00 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.458+0000 7f04fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62973792 0 0) 0x7f05141355f0 con 0x7f05140b4e90 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0500004eb0 con 0x7f05140b4e90 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972934531 0 0) 0x7f05141358d0 con 0x7f0514134020 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:40:24.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.459+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05141355f0 con 0x7f0514134020 2024-03-25T19:40:24.464 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.460+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406148750 0 0) 0x7f0514135380 con 0x7f051413fd00 2024-03-25T19:40:24.464 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.460+0000 7f04fbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.464 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.460+0000 7f04fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.464 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.460+0000 7f04fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.464 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.460+0000 7f04fbfff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:40:24.464 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.460+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05141358d0 con 0x7f051413fd00 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.463+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647465271 0 0) 0x7f0500004eb0 con 0x7f05140b4e90 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.463+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0514135380 con 0x7f05140b4e90 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.463+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0508001c00 con 0x7f05140b4e90 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.463+0000 7f04fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.463+0000 7f04fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.463+0000 7f04fbfff640 10 monclient(hunting): dump: 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.466 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.463+0000 7f04fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.464+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3365313212 0 0) 0x7f05141355f0 con 0x7f0514134020 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.464+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0500004eb0 con 0x7f0514134020 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.464+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f050c001b50 con 0x7f0514134020 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.464+0000 7f04fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.464+0000 7f04fbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.464+0000 7f04fbfff640 10 monclient(hunting): dump: 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.467 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.465+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 547511527 0 0) 0x7f05141358d0 con 0x7f051413fd00 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.465+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05141355f0 con 0x7f051413fd00 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.465+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0504001830 con 0x7f051413fd00 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.465+0000 7f04fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.465+0000 7f04fbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.465+0000 7f04fbfff640 10 monclient(hunting): dump: 2024-03-25T19:40:24.468 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.469 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 120507295 0 0) 0x7f0500004eb0 con 0x7f0514134020 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 >> v1:172.21.15.178:6789/0 conn(0x7f05140b4e90 legacy=0x7f0514141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 >> v1:172.21.15.110:6789/0 conn(0x7f051413fd00 legacy=0x7f0514135cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 1 monclient: found mon.c 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514142c20 con 0x7f0514134020 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 10 monclient: _finish_auth 0 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f04fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 5 monclient: authenticate success, global_id 4151 2024-03-25T19:40:24.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 10 monclient: _renew_subs 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0514134430 con 0x7f0514134020 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 10 monclient: _renew_subs 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05141e3a10 con 0x7f0514134020 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.468+0000 7f0519e63640 1 librados: init done 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.469+0000 7f0519e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.469+0000 7f0519e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.469+0000 7f0519e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.470+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0514132e30 con 0x7f0514134020 2024-03-25T19:40:24.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f050c002060 con 0x7f0514134020 2024-03-25T19:40:24.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:24.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f050c004d00 con 0x7f0514134020 2024-03-25T19:40:24.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 10 monclient: dump: 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f050c0051f0 con 0x7f0514134020 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f0511ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:24.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 136580853 0 0) 0x7f050c0069b0 con 0x7f0514134020 2024-03-25T19:40:24.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 10 client.4151.objecter ms_dispatch 0x7f0514000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:40:24.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 3 client.4151.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:40:24.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.472+0000 7f04fbfff640 3 client.4151.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:40:24.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.473+0000 7f04fbfff640 20 client.4151.objecter dump_active .. 0 homeless 2024-03-25T19:40:24.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.566+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f050c006bc0 con 0x7f0514134020 2024-03-25T19:40:24.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.566+0000 7f04fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.566+0000 7f04fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:24.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.656+0000 7f0519e63640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c08330a9-07e8-49ab-b0c7-fbade08e842d", "id": 5}] 2024-03-25T19:40:24.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.656+0000 7f0519e63640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c08330a9-07e8-49ab-b0c7-fbade08e842d", "id": 5}] 2024-03-25T19:40:24.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.656+0000 7f0519e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.657 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.656+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c08330a9-07e8-49ab-b0c7-fbade08e842d", "id": 5} v 0) v1 -- 0x7f0514066ea0 con 0x7f0514134020 2024-03-25T19:40:24.681 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:24.681+0000 7fb51062b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:40:24.735 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.734+0000 7f04fbfff640 1 -- 172.21.15.178:0/576404234 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c08330a9-07e8-49ab-b0c7-fbade08e842d", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 282467498 0 1988412176) 0x7f050c006610 con 0x7f0514134020 2024-03-25T19:40:24.735 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.735+0000 7f04fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c08330a9-07e8-49ab-b0c7-fbade08e842d", "id": 5}] 2024-03-25T19:40:24.735 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.735+0000 7f04fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:24.735 INFO:teuthology.orchestra.run.smithi178.stdout:5 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.735+0000 7f0519e63640 10 librados: watch_flush enter 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.735+0000 7f0519e63640 10 librados: watch_flush exit 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.735+0000 7f0519e63640 20 client.4151.objecter shutdown clearing up homeless session... 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.735+0000 7f0519e63640 10 client.4151.objecter successfully canceled tick 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.735+0000 7f0519e63640 10 monclient: shutdown 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.735+0000 7f0519e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.736+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 >> v1:172.21.15.110:6790/0 conn(0x7f0514134020 legacy=0x7f05141e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.736+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 shutdown_connections 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.736+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 >> 172.21.15.178:0/576404234 conn(0x7f0514066ac0 msgr2=0x7f05141393d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.736+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 shutdown_connections 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.736+0000 7f0519e63640 1 -- 172.21.15.178:0/576404234 wait complete. 2024-03-25T19:40:24.737 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.736+0000 7f0519e63640 1 librados: shutdown 2024-03-25T19:40:24.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 57e9f7f2-6fb8-4958-9cae-c8218eebd46b 6 2024-03-25T19:40:24.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.847+0000 7fcda5d7b640 10 monclient: get_monmap_and_config 2024-03-25T19:40:24.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.847+0000 7fcda5d7b640 10 monclient: build_initial_monmap 2024-03-25T19:40:24.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.847+0000 7fcda5d7b640 10 monclient: monmap: 2024-03-25T19:40:24.848 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:40:24.849693+0000 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:40:24.849693+0000 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:24.849 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.848+0000 7fcda5d7b640 1 Processor -- start 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.848+0000 7fcda5d7b640 1 -- start start 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.848+0000 7fcda5d7b640 10 monclient: init 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 10 monclient(hunting): picked mon.noname-a con 0x7fcda00b4700 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 10 monclient(hunting): picked mon.noname-b con 0x7fcda0142a20 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 10 monclient(hunting): picked mon.noname-c con 0x7fcda00a96b0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00a9d60 con 0x7fcda00b4700 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda013ff10 con 0x7fcda00a96b0 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00aa1f0 con 0x7fcda0142a20 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:24.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcda5d7b640 10 monclient(hunting): authenticate will time out at 4179.999789s 2024-03-25T19:40:24.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcd9ffff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fcda00a96b0 0x7fcda013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34340/0 (socket says 172.21.15.178:34340) 2024-03-25T19:40:24.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.849+0000 7fcd9ffff640 1 -- 172.21.15.178:0/2649544870 learned_addr learned my addr 172.21.15.178:0/2649544870 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860752277 0 0) 0x7fcda00aa1f0 con 0x7fcda0142a20 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd88003520 con 0x7fcda0142a20 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921289057 0 0) 0x7fcda00a9d60 con 0x7fcda00b4700 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:40:24.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda00aa1f0 con 0x7fcda00b4700 2024-03-25T19:40:24.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110551394 0 0) 0x7fcda013ff10 con 0x7fcda00a96b0 2024-03-25T19:40:24.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:40:24.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.852+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda00a9d60 con 0x7fcda00a96b0 2024-03-25T19:40:24.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30471947 0 0) 0x7fcd88003520 con 0x7fcda0142a20 2024-03-25T19:40:24.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 >> v1:172.21.15.178:6789/0 conn(0x7fcda00a96b0 legacy=0x7fcda013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 >> v1:172.21.15.110:6789/0 conn(0x7fcda00b4700 legacy=0x7fcda0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda00b4ae0 con 0x7fcda0142a20 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcd9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcda5d7b640 5 monclient: authenticate success, global_id 4154 2024-03-25T19:40:24.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.855+0000 7fcda5d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:24.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcd90001650 con 0x7fcda0142a20 2024-03-25T19:40:24.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:24.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 10 monclient: dump: 2024-03-25T19:40:24.857 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.857 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.857 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.857 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd900018a0 con 0x7fcda0142a20 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcda5d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcda5d7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 1 -- 172.21.15.178:0/2649544870 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcd90001c70 con 0x7fcda0142a20 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcda5d7b640 4 set_mon_vals no callback set 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:24.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.856+0000 7fcd9e7fc640 10 monclient: dump: 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.859 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.857+0000 7fcda5d7b640 10 monclient: shutdown 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.857+0000 7fcda5d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.857+0000 7fcda5d7b640 1 -- 172.21.15.178:0/2649544870 >> v1:172.21.15.110:6790/0 conn(0x7fcda0142a20 legacy=0x7fcda013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.857+0000 7fcda5d7b640 1 -- 172.21.15.178:0/2649544870 shutdown_connections 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.857+0000 7fcda5d7b640 1 -- 172.21.15.178:0/2649544870 >> 172.21.15.178:0/2649544870 conn(0x7fcda00668e0 msgr2=0x7fcda0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.857+0000 7fcda5d7b640 1 -- 172.21.15.178:0/2649544870 shutdown_connections 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.857+0000 7fcda5d7b640 1 -- 172.21.15.178:0/2649544870 wait complete. 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 10 monclient: build_initial_monmap 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 10 monclient: monmap: 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 1 librados: starting msgr at 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 1 librados: starting objecter 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 1 Processor -- start 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 1 -- start start 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 1 librados: setting wanted keys 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 1 librados: calling monclient init 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.858+0000 7fcda5d7b640 10 monclient: init 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 10 monclient(hunting): picked mon.noname-b con 0x7fcda00a96b0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 10 monclient(hunting): picked mon.noname-a con 0x7fcda00b4700 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 10 monclient(hunting): picked mon.noname-c con 0x7fcda0142a20 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00abee0 con 0x7fcda00b4700 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda013be80 con 0x7fcda00a96b0 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00acda0 con 0x7fcda0142a20 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcda5d7b640 10 monclient(hunting): authenticate will time out at 4180.009940s 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcd9effd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcda00b4700 0x7fcda013b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.178:50706/0 (socket says 172.21.15.178:50706) 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcd9effd640 1 -- 172.21.15.178:0/4079642252 learned_addr learned my addr 172.21.15.178:0/4079642252 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.859+0000 7fcd9f7fe640 1 --1- 172.21.15.178:0/4079642252 >> v1:172.21.15.178:6789/0 conn(0x7fcda00a96b0 0x7fcda00ac690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34354/0 (socket says 172.21.15.178:34354) 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.860+0000 7fcd87fff640 10 client.?.objecter ms_handle_connect 0x7fcda00b4700 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.860+0000 7fcd87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.860+0000 7fcd87fff640 10 client.?.objecter ms_handle_connect 0x7fcda00a96b0 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.860+0000 7fcd87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.860+0000 7fcd87fff640 10 client.?.objecter ms_handle_connect 0x7fcda0142a20 2024-03-25T19:40:24.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.860+0000 7fcd87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.861+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747237191 0 0) 0x7fcda00acda0 con 0x7fcda0142a20 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd8c004eb0 con 0x7fcda0142a20 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592407342 0 0) 0x7fcda00abee0 con 0x7fcda00b4700 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda00acda0 con 0x7fcda00b4700 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404490607 0 0) 0x7fcda013be80 con 0x7fcda00a96b0 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:24.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:24.867 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:24.867 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:40:24.867 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.862+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda00abee0 con 0x7fcda00a96b0 2024-03-25T19:40:24.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838734306 0 0) 0x7fcd8c004eb0 con 0x7fcda0142a20 2024-03-25T19:40:24.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcda013be80 con 0x7fcda0142a20 2024-03-25T19:40:24.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3526211779 0 0) 0x7fcda00acda0 con 0x7fcda00b4700 2024-03-25T19:40:24.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd8c004eb0 con 0x7fcda00b4700 2024-03-25T19:40:24.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcd90001bb0 con 0x7fcda00b4700 2024-03-25T19:40:24.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:24.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 10 monclient(hunting): dump: 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2355192804 0 0) 0x7fcda00abee0 con 0x7fcda00a96b0 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcda00acda0 con 0x7fcda00a96b0 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcd940017e0 con 0x7fcda00a96b0 2024-03-25T19:40:24.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.866+0000 7fcd87fff640 10 monclient(hunting): dump: 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.870 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2774022985 0 0) 0x7fcd8c004eb0 con 0x7fcda00b4700 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 >> v1:172.21.15.110:6790/0 conn(0x7fcda0142a20 legacy=0x7fcda01e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 >> v1:172.21.15.178:6789/0 conn(0x7fcda00a96b0 legacy=0x7fcda00ac690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 1 monclient: found mon.a 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda00acf30 con 0x7fcda00b4700 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcd87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 5 monclient: authenticate success, global_id 4155 2024-03-25T19:40:24.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 10 monclient: _renew_subs 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda013ba50 con 0x7fcda00b4700 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 10 monclient: _renew_subs 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda0139ec0 con 0x7fcda00b4700 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.869+0000 7fcda5d7b640 1 librados: init done 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.870+0000 7fcda5d7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.870+0000 7fcda5d7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.870+0000 7fcda5d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:24.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.870+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda013bce0 con 0x7fcda00b4700 2024-03-25T19:40:24.967 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:24.966+0000 7fb51062b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:40:24.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.968+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd90001230 con 0x7fcda00b4700 2024-03-25T19:40:24.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.968+0000 7fcd87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:24.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.968+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcd90003960 con 0x7fcda00b4700 2024-03-25T19:40:24.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.968+0000 7fcd87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.969+0000 7fcd87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:24.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.969+0000 7fcd87fff640 10 monclient: dump: 2024-03-25T19:40:24.969 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:24.969 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.969+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcd90003f10 con 0x7fcda00b4700 2024-03-25T19:40:24.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.969+0000 7fcd9dffb640 4 set_mon_vals no callback set 2024-03-25T19:40:24.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.971+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1441090758 0 0) 0x7fcd90006870 con 0x7fcda00b4700 2024-03-25T19:40:24.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.971+0000 7fcd87fff640 10 client.4155.objecter ms_dispatch 0x7fcda0000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:40:24.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.971+0000 7fcd87fff640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:40:24.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.971+0000 7fcd87fff640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:40:24.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.971+0000 7fcd87fff640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T19:40:24.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.971+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcd900043a0 con 0x7fcda00b4700 2024-03-25T19:40:24.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.971+0000 7fcd87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:24.971+0000 7fcd87fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:25.022 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:25.022+0000 7fb51062b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:40:25.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.043+0000 7fcda5d7b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "57e9f7f2-6fb8-4958-9cae-c8218eebd46b", "id": 6}] 2024-03-25T19:40:25.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.043+0000 7fcda5d7b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "57e9f7f2-6fb8-4958-9cae-c8218eebd46b", "id": 6}] 2024-03-25T19:40:25.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.043+0000 7fcda5d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:25.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.043+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd new", "uuid": "57e9f7f2-6fb8-4958-9cae-c8218eebd46b", "id": 6} v 0) v1 -- 0x7fcda004f730 con 0x7fcda00b4700 2024-03-25T19:40:25.091 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.090+0000 7fcd87fff640 1 -- 172.21.15.178:0/4079642252 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "57e9f7f2-6fb8-4958-9cae-c8218eebd46b", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 513952696 0 1113811849) 0x7fcd90006b60 con 0x7fcda00b4700 2024-03-25T19:40:25.091 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.090+0000 7fcd87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "57e9f7f2-6fb8-4958-9cae-c8218eebd46b", "id": 6}] 2024-03-25T19:40:25.091 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.090+0000 7fcd87fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:25.091 INFO:teuthology.orchestra.run.smithi178.stdout:6 2024-03-25T19:40:25.092 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 10 librados: watch_flush enter 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 10 librados: watch_flush exit 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 10 client.4155.objecter successfully canceled tick 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 10 monclient: shutdown 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 >> v1:172.21.15.110:6789/0 conn(0x7fcda00b4700 legacy=0x7fcda013b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 shutdown_connections 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.091+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 >> 172.21.15.178:0/4079642252 conn(0x7fcda00668e0 msgr2=0x7fcda00b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.092+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 shutdown_connections 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.092+0000 7fcda5d7b640 1 -- 172.21.15.178:0/4079642252 wait complete. 2024-03-25T19:40:25.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.092+0000 7fcda5d7b640 1 librados: shutdown 2024-03-25T19:40:25.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 9e856141-e93f-4a86-bcbc-2e6e5778e31c 7 2024-03-25T19:40:25.143 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:25.143+0000 7fb51062b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:40:25.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.200+0000 7ff010243640 10 monclient: get_monmap_and_config 2024-03-25T19:40:25.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.200+0000 7ff010243640 10 monclient: build_initial_monmap 2024-03-25T19:40:25.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.200+0000 7ff010243640 10 monclient: monmap: 2024-03-25T19:40:25.200 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:25.200 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.201 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:40:25.201902+0000 2024-03-25T19:40:25.201 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:40:25.201902+0000 2024-03-25T19:40:25.201 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:25.201 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:25.201 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:25.201 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:25.201 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:25.201 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:25.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.200+0000 7ff010243640 1 Processor -- start 2024-03-25T19:40:25.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.200+0000 7ff010243640 1 -- start start 2024-03-25T19:40:25.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.200+0000 7ff010243640 10 monclient: init 2024-03-25T19:40:25.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 10 monclient(hunting): picked mon.noname-c con 0x7ff0080b4700 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 10 monclient(hunting): picked mon.noname-a con 0x7ff008142a20 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 10 monclient(hunting): picked mon.noname-b con 0x7ff0080a96b0 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0080a9d60 con 0x7ff008142a20 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00813ff10 con 0x7ff0080b4700 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0080aa1f0 con 0x7ff0080a96b0 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.201+0000 7ff010243640 10 monclient(hunting): authenticate will time out at 4180.352033s 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.202+0000 7ff00dfdf640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff0080b4700 0x7ff008142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34358/0 (socket says 172.21.15.178:34358) 2024-03-25T19:40:25.203 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.202+0000 7ff00dfdf640 1 -- 172.21.15.178:0/1551892488 learned_addr learned my addr 172.21.15.178:0/1551892488 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:25.204 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725103500 0 0) 0x7ff00813ff10 con 0x7ff0080b4700 2024-03-25T19:40:25.204 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff4003520 con 0x7ff0080b4700 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705256328 0 0) 0x7ff0080aa1f0 con 0x7ff0080a96b0 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:40:25.205 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.204+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff00813ff10 con 0x7ff0080a96b0 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315689693 0 0) 0x7feff4003520 con 0x7ff0080b4700 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 >> v1:172.21.15.110:6790/0 conn(0x7ff0080a96b0 legacy=0x7ff00813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 >> v1:172.21.15.110:6789/0 conn(0x7ff008142a20 legacy=0x7ff00813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 1 monclient: found mon.noname-c 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0080b4ae0 con 0x7ff0080b4700 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 10 monclient: _finish_auth 0 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff004001690 con 0x7ff0080b4700 2024-03-25T19:40:25.208 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff00cfdd640 10 monclient: dump: 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff010243640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:40:25.209 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.207+0000 7ff010243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:25.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0040017f0 con 0x7ff0080b4700 2024-03-25T19:40:25.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff00cfdd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:25.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff00cfdd640 1 -- 172.21.15.178:0/1551892488 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff004001c00 con 0x7ff0080b4700 2024-03-25T19:40:25.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff00cfdd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff00cfdd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:25.212 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff00cfdd640 10 monclient: dump: 2024-03-25T19:40:25.212 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 4 set_mon_vals no callback set 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 10 monclient: shutdown 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.213 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 1 -- 172.21.15.178:0/1551892488 >> v1:172.21.15.178:6789/0 conn(0x7ff0080b4700 legacy=0x7ff008142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 1 -- 172.21.15.178:0/1551892488 shutdown_connections 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 1 -- 172.21.15.178:0/1551892488 >> 172.21.15.178:0/1551892488 conn(0x7ff0080668e0 msgr2=0x7ff008066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 1 -- 172.21.15.178:0/1551892488 shutdown_connections 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.209+0000 7ff010243640 1 -- 172.21.15.178:0/1551892488 wait complete. 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 10 monclient: build_initial_monmap 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 10 monclient: monmap: 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 0 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 0.000000 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:created 0.000000 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 1 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:25.214 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 1 librados: starting msgr at 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 1 librados: starting objecter 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 1 Processor -- start 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 1 -- start start 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 1 librados: setting wanted keys 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 1 librados: calling monclient init 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.210+0000 7ff010243640 10 monclient: init 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 10 monclient(hunting): picked mon.noname-b con 0x7ff0080a96b0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 10 monclient(hunting): picked mon.noname-a con 0x7ff0080b4700 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 10 monclient(hunting): picked mon.noname-c con 0x7ff008142a20 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff008136900 con 0x7ff0080b4700 2024-03-25T19:40:25.215 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0081d74d0 con 0x7ff0080a96b0 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff008137eb0 con 0x7ff008142a20 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff010243640 10 monclient(hunting): authenticate will time out at 4180.361839s 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff00dfdf640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff0080a96b0 0x7ff0081361f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:34366/0 (socket says 172.21.15.178:34366) 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.211+0000 7ff00dfdf640 1 -- 172.21.15.178:0/253520077 learned_addr learned my addr 172.21.15.178:0/253520077 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.212+0000 7feffe7fc640 10 client.?.objecter ms_handle_connect 0x7ff0080a96b0 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.212+0000 7feffe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.212+0000 7feffe7fc640 10 client.?.objecter ms_handle_connect 0x7ff008142a20 2024-03-25T19:40:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.212+0000 7feffe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.213+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559533538 0 0) 0x7ff0081d74d0 con 0x7ff0080a96b0 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.213+0000 7feffe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.213+0000 7feffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.213+0000 7feffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.213+0000 7feffe7fc640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.213+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefec004eb0 con 0x7ff0080a96b0 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.214+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885035811 0 0) 0x7ff008137eb0 con 0x7ff008142a20 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.214+0000 7feffe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.217 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.214+0000 7feffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.214+0000 7feffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.214+0000 7feffe7fc640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:40:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.214+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0081d74d0 con 0x7ff008142a20 2024-03-25T19:40:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2123348280 0 0) 0x7fefec004eb0 con 0x7ff0080a96b0 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff008137eb0 con 0x7ff0080a96b0 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff004001c00 con 0x7ff0080a96b0 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:25.219 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350691125 0 0) 0x7ff0081d74d0 con 0x7ff008142a20 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefec004eb0 con 0x7ff008142a20 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff000001b50 con 0x7ff008142a20 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.218+0000 7feffe7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:25.220 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:25.221 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:25.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 840529781 0 0) 0x7ff008137eb0 con 0x7ff0080a96b0 2024-03-25T19:40:25.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 >> v1:172.21.15.110:6790/0 conn(0x7ff008142a20 legacy=0x7ff00812ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 >> v1:172.21.15.110:6789/0 conn(0x7ff0080b4700 legacy=0x7ff0081d7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:25.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 1 monclient: found mon.b 2024-03-25T19:40:25.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:25.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:25.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0081d3fd0 con 0x7ff0080a96b0 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7feffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 5 monclient: authenticate success, global_id 4162 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 10 monclient: _renew_subs 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00813bc10 con 0x7ff0080a96b0 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 10 monclient: _renew_subs 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff008137030 con 0x7ff0080a96b0 2024-03-25T19:40:25.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.221+0000 7ff010243640 1 librados: init done 2024-03-25T19:40:25.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.222+0000 7ff010243640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.222+0000 7ff010243640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.222+0000 7ff010243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:25.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.222+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0081d7680 con 0x7ff0080a96b0 2024-03-25T19:40:25.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff004004a90 con 0x7ff0080a96b0 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff004004e20 con 0x7ff0080a96b0 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 10 monclient: dump: 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:epoch 1 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:election_strategy: 3 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:40:25.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff0040053d0 con 0x7ff0080a96b0 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffffff640 4 set_mon_vals no callback set 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2489415860 0 0) 0x7ff0040064e0 con 0x7ff0080a96b0 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 10 client.4162.objecter ms_dispatch 0x7ff008000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.315+0000 7feffe7fc640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.316+0000 7feffe7fc640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.316+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff004018dc0 con 0x7ff0080a96b0 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.316+0000 7feffe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.316+0000 7feffe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:25.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:25.317+0000 7f4bc9813040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:40:25.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.385+0000 7ff010243640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9e856141-e93f-4a86-bcbc-2e6e5778e31c", "id": 7}] 2024-03-25T19:40:25.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.385+0000 7ff010243640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9e856141-e93f-4a86-bcbc-2e6e5778e31c", "id": 7}] 2024-03-25T19:40:25.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.385+0000 7ff010243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:25.386 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.385+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9e856141-e93f-4a86-bcbc-2e6e5778e31c", "id": 7} v 0) v1 -- 0x7ff00804f730 con 0x7ff0080a96b0 2024-03-25T19:40:25.526 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.526+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1583291820 0 0) 0x7ff004006860 con 0x7ff0080a96b0 2024-03-25T19:40:25.628 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:25.629+0000 7f4bc9813040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:40:25.684 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:25.685+0000 7f4bc9813040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:40:25.743 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.743+0000 7feffe7fc640 1 -- 172.21.15.178:0/253520077 <== mon.1 v1:172.21.15.178:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e856141-e93f-4a86-bcbc-2e6e5778e31c", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 4009329979 0 1371665406) 0x7ff004015f40 con 0x7ff0080a96b0 2024-03-25T19:40:25.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.743+0000 7feffe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9e856141-e93f-4a86-bcbc-2e6e5778e31c", "id": 7}] 2024-03-25T19:40:25.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.743+0000 7feffe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:25.744 INFO:teuthology.orchestra.run.smithi178.stdout:7 2024-03-25T19:40:25.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 10 librados: watch_flush enter 2024-03-25T19:40:25.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 10 librados: watch_flush exit 2024-03-25T19:40:25.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T19:40:25.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 10 client.4162.objecter successfully canceled tick 2024-03-25T19:40:25.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 10 monclient: shutdown 2024-03-25T19:40:25.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 >> v1:172.21.15.178:6789/0 conn(0x7ff0080a96b0 legacy=0x7ff0081361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.745 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 shutdown_connections 2024-03-25T19:40:25.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 >> 172.21.15.178:0/253520077 conn(0x7ff0080668e0 msgr2=0x7ff0080b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 shutdown_connections 2024-03-25T19:40:25.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 1 -- 172.21.15.178:0/253520077 wait complete. 2024-03-25T19:40:25.746 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-25T19:40:25.744+0000 7ff010243640 1 librados: shutdown 2024-03-25T19:40:25.754 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:25.755 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:40:25.755 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:40:25.757 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:40:25.757 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:25.757 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:40:25.757 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:40:25.759 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:40:25.760 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:25.760 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:40:25.760 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:40:25.762 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:40:25.762 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:25.762 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:40:25.762 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:40:25.764 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:40:25.764 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:25.764 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:40:25.764 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:40:25.798 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:40:25.798 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:25.802 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:40:25.802 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:40:25.804 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:40:25.804 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:25.804 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:40:25.804 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:40:25.806 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:40:25.806 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:25.807 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:40:25.807 DEBUG:teuthology.orchestra.run.smithi178:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:40:25.809 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:40:25.809 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:25.841 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-03-25T19:40:25.842+0000 7f4bc9813040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:40:25.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.968+0000 7fba554ba640 10 monclient: get_monmap_and_config 2024-03-25T19:40:25.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.968+0000 7fba554ba640 10 monclient: build_initial_monmap 2024-03-25T19:40:25.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.968+0000 7fba554ba640 10 monclient: monmap: 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:40:25.969278+0000 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:40:25.969278+0000 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:25.968 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:25.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 1 Processor -- start 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 1 -- start start 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 10 monclient: init 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 10 monclient(hunting): picked mon.noname-b con 0x7fba500a9920 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 10 monclient(hunting): picked mon.noname-a con 0x7fba500b45a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 10 monclient(hunting): picked mon.noname-c con 0x7fba50146450 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5006b100 con 0x7fba500b45a0 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba500b2940 con 0x7fba50146450 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5013fec0 con 0x7fba500a9920 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.969+0000 7fba554ba640 10 monclient(hunting): authenticate will time out at 4197.110790s 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.970+0000 7fba4effd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fba500a9920 0x7fba5013f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33710/0 (socket says 172.21.15.110:33710) 2024-03-25T19:40:25.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.970+0000 7fba4effd640 1 -- 172.21.15.110:0/1584665732 learned_addr learned my addr 172.21.15.110:0/1584665732 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.973+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567897098 0 0) 0x7fba5013fec0 con 0x7fba500a9920 2024-03-25T19:40:25.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.973+0000 7fba4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.973+0000 7fba4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.973+0000 7fba4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.973+0000 7fba4dffb640 10 monclient(hunting): my global_id is 4178 2024-03-25T19:40:25.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.973+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba38003520 con 0x7fba500a9920 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 719233014 0 0) 0x7fba38003520 con 0x7fba500a9920 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 >> v1:172.21.15.178:6789/0 conn(0x7fba50146450 legacy=0x7fba5014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 >> v1:172.21.15.110:6789/0 conn(0x7fba500b45a0 legacy=0x7fba50145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 1 monclient: found mon.noname-b 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50140050 con 0x7fba500a9920 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fba40001690 con 0x7fba500a9920 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba4dffb640 10 monclient: dump: 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba554ba640 5 monclient: authenticate success, global_id 4178 2024-03-25T19:40:25.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.977+0000 7fba554ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.983+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba400017f0 con 0x7fba500a9920 2024-03-25T19:40:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.983+0000 7fba4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.983+0000 7fba4dffb640 1 -- 172.21.15.110:0/1584665732 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fba40001aa0 con 0x7fba500a9920 2024-03-25T19:40:25.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.983+0000 7fba4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.983+0000 7fba4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.983+0000 7fba4dffb640 10 monclient: dump: 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.983+0000 7fba554ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.983+0000 7fba554ba640 4 set_mon_vals no callback set 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.984+0000 7fba554ba640 10 monclient: shutdown 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.984+0000 7fba554ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.984+0000 7fba554ba640 1 -- 172.21.15.110:0/1584665732 >> v1:172.21.15.110:6790/0 conn(0x7fba500a9920 legacy=0x7fba5013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.984+0000 7fba554ba640 1 -- 172.21.15.110:0/1584665732 shutdown_connections 2024-03-25T19:40:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.984+0000 7fba554ba640 1 -- 172.21.15.110:0/1584665732 >> 172.21.15.110:0/1584665732 conn(0x7fba500b7f40 msgr2=0x7fba500b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.985+0000 7fba554ba640 1 -- 172.21.15.110:0/1584665732 shutdown_connections 2024-03-25T19:40:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.985+0000 7fba554ba640 1 -- 172.21.15.110:0/1584665732 wait complete. 2024-03-25T19:40:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.986+0000 7fba554ba640 10 monclient: build_initial_monmap 2024-03-25T19:40:25.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.986+0000 7fba554ba640 10 monclient: monmap: 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:25.986 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:25.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.986+0000 7fba554ba640 1 librados: starting msgr at 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.987+0000 7fba554ba640 1 librados: starting objecter 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.987+0000 7fba554ba640 1 Processor -- start 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.987+0000 7fba554ba640 1 -- start start 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.987+0000 7fba554ba640 1 librados: setting wanted keys 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.987+0000 7fba554ba640 1 librados: calling monclient init 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.987+0000 7fba554ba640 10 monclient: init 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 10 monclient(hunting): picked mon.noname-c con 0x7fba500a9920 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 10 monclient(hunting): picked mon.noname-a con 0x7fba500b45a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 10 monclient(hunting): picked mon.noname-b con 0x7fba50146450 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba50142a50 con 0x7fba500b45a0 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba50142f70 con 0x7fba50146450 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba50141ee0 con 0x7fba500a9920 2024-03-25T19:40:25.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba554ba640 10 monclient(hunting): authenticate will time out at 4197.129039s 2024-03-25T19:40:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba4effd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fba500a9920 0x7fba50135030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33718/0 (socket says 172.21.15.110:33718) 2024-03-25T19:40:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.988+0000 7fba4effd640 1 -- 172.21.15.110:0/3099323474 learned_addr learned my addr 172.21.15.110:0/3099323474 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.989+0000 7fba377fe640 10 client.?.objecter ms_handle_connect 0x7fba500a9920 2024-03-25T19:40:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.989+0000 7fba377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.989+0000 7fba377fe640 10 client.?.objecter ms_handle_connect 0x7fba50146450 2024-03-25T19:40:25.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:25.989+0000 7fba377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:26.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.073+0000 7fba377fe640 10 client.?.objecter ms_handle_connect 0x7fba500b45a0 2024-03-25T19:40:26.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.073+0000 7fba377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:26.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.086+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25005863 0 0) 0x7fba50142a50 con 0x7fba500b45a0 2024-03-25T19:40:26.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.086+0000 7fba377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:26.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.086+0000 7fba377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:26.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.086+0000 7fba377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:26.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.086+0000 7fba377fe640 10 monclient(hunting): my global_id is 4164 2024-03-25T19:40:26.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.086+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba3c004eb0 con 0x7fba500b45a0 2024-03-25T19:40:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.097+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 937339538 0 0) 0x7fba3c004eb0 con 0x7fba500b45a0 2024-03-25T19:40:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.097+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba50142a50 con 0x7fba500b45a0 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.098+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fba44001a10 con 0x7fba500b45a0 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.098+0000 7fba377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.098+0000 7fba377fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.098+0000 7fba377fe640 10 monclient(hunting): dump: 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:26.097 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:26.098 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:26.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.098+0000 7fba377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.100+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222488329 0 0) 0x7fba50142f70 con 0x7fba50146450 2024-03-25T19:40:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.100+0000 7fba377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.100+0000 7fba377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:26.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.100+0000 7fba377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:26.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.100+0000 7fba377fe640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:40:26.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.100+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba3c004eb0 con 0x7fba50146450 2024-03-25T19:40:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1796696251 0 0) 0x7fba50142a50 con 0x7fba500b45a0 2024-03-25T19:40:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 >> v1:172.21.15.110:6790/0 conn(0x7fba500a9920 legacy=0x7fba50135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 >> v1:172.21.15.178:6789/0 conn(0x7fba50146450 legacy=0x7fba501e2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 1 monclient: found mon.a 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50142070 con 0x7fba500b45a0 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 5 monclient: authenticate success, global_id 4164 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 10 monclient: _renew_subs 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba50135740 con 0x7fba500b45a0 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 10 client.4164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 10 monclient: _renew_subs 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5013be90 con 0x7fba500b45a0 2024-03-25T19:40:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.103+0000 7fba554ba640 1 librados: init done 2024-03-25T19:40:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.104+0000 7fba554ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.104+0000 7fba554ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.104+0000 7fba554ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.104+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba501441a0 con 0x7fba500b45a0 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.227+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba44003690 con 0x7fba500b45a0 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.227+0000 7fba377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.227+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fba44003900 con 0x7fba500b45a0 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.227+0000 7fba377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.227+0000 7fba377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.227+0000 7fba377fe640 10 monclient: dump: 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:26.228 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:40:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.227+0000 7fba4d7fa640 4 set_mon_vals no callback set 2024-03-25T19:40:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.228+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1583291820 0 0) 0x7fba44003b20 con 0x7fba500b45a0 2024-03-25T19:40:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.238+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fba440143f0 con 0x7fba500b45a0 2024-03-25T19:40:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.238+0000 7fba377fe640 10 client.4164.objecter ms_dispatch 0x7fba50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.238+0000 7fba377fe640 3 client.4164.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.238+0000 7fba377fe640 3 client.4164.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.238+0000 7fba377fe640 20 client.4164.objecter dump_active .. 0 homeless 2024-03-25T19:40:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.238+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fba440136b0 con 0x7fba500b45a0 2024-03-25T19:40:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.238+0000 7fba377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.238+0000 7fba377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:26.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.338+0000 7fba554ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:26.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.338+0000 7fba554ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:26.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.338+0000 7fba554ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:26.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.338+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba50142260 con 0x7fba500b45a0 2024-03-25T19:40:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.435+0000 7fba377fe640 1 -- 172.21.15.110:0/3099323474 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fba44011e90 con 0x7fba500b45a0 2024-03-25T19:40:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.435+0000 7fba377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:26.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.435+0000 7fba377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:26.435 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-25T19:40:26.435 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:26.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 10 librados: watch_flush enter 2024-03-25T19:40:26.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 10 librados: watch_flush exit 2024-03-25T19:40:26.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 20 client.4164.objecter shutdown clearing up homeless session... 2024-03-25T19:40:26.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 10 client.4164.objecter successfully canceled tick 2024-03-25T19:40:26.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 10 monclient: shutdown 2024-03-25T19:40:26.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:26.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 >> v1:172.21.15.110:6789/0 conn(0x7fba500b45a0 legacy=0x7fba50144480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:26.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 shutdown_connections 2024-03-25T19:40:26.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 >> 172.21.15.110:0/3099323474 conn(0x7fba500b7f40 msgr2=0x7fba50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:26.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 shutdown_connections 2024-03-25T19:40:26.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 1 -- 172.21.15.110:0/3099323474 wait complete. 2024-03-25T19:40:26.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-03-25T19:40:26.436+0000 7fba554ba640 1 librados: shutdown 2024-03-25T19:40:26.449 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:40:26.449 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:40:27.169 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:27.169+0000 7fb4b0bd5640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:40:27.192 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:40:27.192+0000 7fb4b0bd5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:40:27.254 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:27.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.403+0000 7f046ec8d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:27.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.403+0000 7f046ec8d640 10 monclient: build_initial_monmap 2024-03-25T19:40:27.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.403+0000 7f046ec8d640 10 monclient: monmap: 2024-03-25T19:40:27.402 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:40:27.403953+0000 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:40:27.403953+0000 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:27.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.404+0000 7f046ec8d640 1 Processor -- start 2024-03-25T19:40:27.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.404+0000 7f046ec8d640 1 -- start start 2024-03-25T19:40:27.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.404+0000 7f046ec8d640 10 monclient: init 2024-03-25T19:40:27.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:27.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:27.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 10 monclient(hunting): picked mon.noname-a con 0x7f04680b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:27.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 10 monclient(hunting): picked mon.noname-b con 0x7f04680b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:27.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 10 monclient(hunting): picked mon.noname-c con 0x7f046814be90 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:27.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680af940 con 0x7f04680b4d60 2024-03-25T19:40:27.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680b0a70 con 0x7f046814be90 2024-03-25T19:40:27.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680b0d50 con 0x7f04680b1730 2024-03-25T19:40:27.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:27.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.405+0000 7f046ec8d640 10 monclient(hunting): authenticate will time out at 4198.546537s 2024-03-25T19:40:27.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.406+0000 7f0467fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f04680b1730 0x7f04680b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33732/0 (socket says 172.21.15.110:33732) 2024-03-25T19:40:27.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.406+0000 7f0467fff640 1 -- 172.21.15.110:0/20010244 learned_addr learned my addr 172.21.15.110:0/20010244 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:27.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.409+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048008659 0 0) 0x7f04680b0d50 con 0x7f04680b1730 2024-03-25T19:40:27.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.409+0000 7f04677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:27.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.409+0000 7f04677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.409+0000 7f04677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): my global_id is 4184 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0458003520 con 0x7f04680b1730 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900427247 0 0) 0x7f04680af940 con 0x7f04680b4d60 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04680b0d50 con 0x7f04680b4d60 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124844310 0 0) 0x7f04680b0a70 con 0x7f046814be90 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 10 monclient(hunting): my global_id is 4180 2024-03-25T19:40:27.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.410+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04680af940 con 0x7f046814be90 2024-03-25T19:40:27.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.414+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3485364837 0 0) 0x7f0458003520 con 0x7f04680b1730 2024-03-25T19:40:27.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 >> v1:172.21.15.178:6789/0 conn(0x7f046814be90 legacy=0x7f046814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:27.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 >> v1:172.21.15.110:6789/0 conn(0x7f04680b4d60 legacy=0x7f046813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:27.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:27.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 1 monclient: found mon.noname-b 2024-03-25T19:40:27.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680b1030 con 0x7f04680b1730 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0454001690 con 0x7f04680b1730 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f04677fe640 10 monclient: dump: 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:27.415 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:27.416 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:27.416 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:27.416 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:27.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f046ec8d640 5 monclient: authenticate success, global_id 4184 2024-03-25T19:40:27.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.415+0000 7f046ec8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.417+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04540018b0 con 0x7f04680b1730 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.417+0000 7f04677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.417+0000 7f04677fe640 1 -- 172.21.15.110:0/20010244 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0454001c40 con 0x7f04680b1730 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.417+0000 7f04677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.417+0000 7f04677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.417+0000 7f04677fe640 10 monclient: dump: 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:27.417 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:27.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.417+0000 7f046ec8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:27.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.417+0000 7f046ec8d640 4 set_mon_vals no callback set 2024-03-25T19:40:27.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.418+0000 7f046ec8d640 10 monclient: shutdown 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.418+0000 7f046ec8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.418+0000 7f046ec8d640 1 -- 172.21.15.110:0/20010244 >> v1:172.21.15.110:6790/0 conn(0x7f04680b1730 legacy=0x7f04680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.418+0000 7f046ec8d640 1 -- 172.21.15.110:0/20010244 shutdown_connections 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.418+0000 7f046ec8d640 1 -- 172.21.15.110:0/20010244 >> 172.21.15.110:0/20010244 conn(0x7f04680617a0 msgr2=0x7f0468061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.418+0000 7f046ec8d640 1 -- 172.21.15.110:0/20010244 shutdown_connections 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.418+0000 7f046ec8d640 1 -- 172.21.15.110:0/20010244 wait complete. 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.419+0000 7f046ec8d640 10 monclient: build_initial_monmap 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.419+0000 7f046ec8d640 10 monclient: monmap: 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:27.419 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:27.420 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:27.420 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:27.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.420+0000 7f046ec8d640 1 librados: starting msgr at 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.420+0000 7f046ec8d640 1 librados: starting objecter 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.420+0000 7f046ec8d640 1 Processor -- start 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.420+0000 7f046ec8d640 1 -- start start 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.420+0000 7f046ec8d640 1 librados: setting wanted keys 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.420+0000 7f046ec8d640 1 librados: calling monclient init 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.420+0000 7f046ec8d640 10 monclient: init 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.421+0000 7f046ec8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.421+0000 7f046ec8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.421+0000 7f046ec8d640 10 monclient(hunting): picked mon.noname-a con 0x7f04680b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046ec8d640 10 monclient(hunting): picked mon.noname-b con 0x7f04680b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046ec8d640 10 monclient(hunting): picked mon.noname-c con 0x7f046814be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046ec8d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0468141bf0 con 0x7f04680b1730 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046ec8d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0468141ed0 con 0x7f04680b4d60 2024-03-25T19:40:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046ec8d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04681421b0 con 0x7f046814be90 2024-03-25T19:40:27.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046ec8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:27.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046ec8d640 10 monclient(hunting): authenticate will time out at 4198.563490s 2024-03-25T19:40:27.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046d22a640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f046814be90 0x7f04681341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33748/0 (socket says 172.21.15.110:33748) 2024-03-25T19:40:27.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.422+0000 7f046d22a640 1 -- 172.21.15.110:0/2810557898 learned_addr learned my addr 172.21.15.110:0/2810557898 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:27.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.423+0000 7f04657fa640 10 client.?.objecter ms_handle_connect 0x7f046814be90 2024-03-25T19:40:27.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.423+0000 7f04657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:27.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.423+0000 7f04657fa640 10 client.?.objecter ms_handle_connect 0x7f04680b4d60 2024-03-25T19:40:27.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.423+0000 7f04657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:27.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.423+0000 7f04657fa640 10 client.?.objecter ms_handle_connect 0x7f04680b1730 2024-03-25T19:40:27.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.423+0000 7f04657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540048887 0 0) 0x7f0468141bf0 con 0x7f04680b1730 2024-03-25T19:40:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 10 monclient(hunting): my global_id is 4170 2024-03-25T19:40:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f044c004eb0 con 0x7f04680b1730 2024-03-25T19:40:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918838386 0 0) 0x7f0468141ed0 con 0x7f04680b4d60 2024-03-25T19:40:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 10 monclient(hunting): my global_id is 4183 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0468141bf0 con 0x7f04680b4d60 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.426+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278236550 0 0) 0x7f04681421b0 con 0x7f046814be90 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.427+0000 7f04657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.427+0000 7f04657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.427+0000 7f04657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.427+0000 7f04657fa640 10 monclient(hunting): my global_id is 4187 2024-03-25T19:40:27.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.427+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0468141ed0 con 0x7f046814be90 2024-03-25T19:40:27.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.433+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3488398693 0 0) 0x7f0468141bf0 con 0x7f04680b4d60 2024-03-25T19:40:27.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.433+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04681421b0 con 0x7f04680b4d60 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0454001670 con 0x7f04680b4d60 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): dump: 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:27.435 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1853339595 0 0) 0x7f044c004eb0 con 0x7f04680b1730 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0468141bf0 con 0x7f04680b1730 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f045c0018b0 con 0x7f04680b1730 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): dump: 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:27.436 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304443 0 0) 0x7f0468141ed0 con 0x7f046814be90 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f044c004eb0 con 0x7f046814be90 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0460001b20 con 0x7f046814be90 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.434+0000 7f04657fa640 10 monclient(hunting): dump: 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:27.437 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:27.438 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3741628306 0 0) 0x7f04681421b0 con 0x7f04680b4d60 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 >> v1:172.21.15.110:6790/0 conn(0x7f046814be90 legacy=0x7f04681341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 >> v1:172.21.15.110:6789/0 conn(0x7f04680b1730 legacy=0x7f04681425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 1 monclient: found mon.b 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468140f00 con 0x7f04680b4d60 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f04657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:27.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 5 monclient: authenticate success, global_id 4183 2024-03-25T19:40:27.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 10 monclient: _renew_subs 2024-03-25T19:40:27.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:27.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04681406f0 con 0x7f04680b4d60 2024-03-25T19:40:27.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:27.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 10 monclient: _renew_subs 2024-03-25T19:40:27.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:27.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0468140380 con 0x7f04680b4d60 2024-03-25T19:40:27.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.439+0000 7f046ec8d640 1 librados: init done 2024-03-25T19:40:27.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.441+0000 7f046ec8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:27.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.441+0000 7f046ec8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:27.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.441+0000 7f046ec8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:27.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.441+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04681401f0 con 0x7f04680b4d60 2024-03-25T19:40:27.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.467+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0454004a90 con 0x7f04680b4d60 2024-03-25T19:40:27.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.467+0000 7f04657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:27.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.467+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0454004da0 con 0x7f04680b4d60 2024-03-25T19:40:27.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.467+0000 7f04657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:27.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.467+0000 7f04657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.467+0000 7f04657fa640 10 monclient: dump: 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.467+0000 7f0466ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:27.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.469+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 383683917 0 0) 0x7f045401ad00 con 0x7f04680b4d60 2024-03-25T19:40:27.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.470+0000 7f046ca29640 10 monclient: get_auth_request con 0x7f044c06e000 auth_method 0 2024-03-25T19:40:27.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.470+0000 7f04657fa640 10 client.4183.objecter ms_handle_connect 0x7f044c06e000 2024-03-25T19:40:27.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.472+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f04540548f0 con 0x7f04680b4d60 2024-03-25T19:40:27.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.472+0000 7f04657fa640 10 client.4183.objecter ms_dispatch 0x7f0468000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:27.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.472+0000 7f04657fa640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:27.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.472+0000 7f04657fa640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:27.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.472+0000 7f04657fa640 20 client.4183.objecter dump_active .. 0 homeless 2024-03-25T19:40:27.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.901+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0454050400 con 0x7f04680b4d60 2024-03-25T19:40:27.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.901+0000 7f04657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:27.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:27.901+0000 7f04657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:28.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.131+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f045404feb0 con 0x7f04680b4d60 2024-03-25T19:40:28.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.259+0000 7f046ec8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:28.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.259+0000 7f046ec8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:28.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.260+0000 7f046ec8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:28.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.260+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0468140880 con 0x7f04680b4d60 2024-03-25T19:40:28.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.327+0000 7f04657fa640 1 -- 172.21.15.110:0/2810557898 <== mon.1 v1:172.21.15.178:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f04540014d0 con 0x7f04680b4d60 2024-03-25T19:40:28.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.327+0000 7f04657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:28.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.327+0000 7f04657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:28.327 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:40:28.327 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:28.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 10 librados: watch_flush enter 2024-03-25T19:40:28.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 10 librados: watch_flush exit 2024-03-25T19:40:28.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 20 client.4183.objecter shutdown clearing up homeless session... 2024-03-25T19:40:28.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 10 client.4183.objecter successfully canceled tick 2024-03-25T19:40:28.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 >> v1:172.21.15.178:6800/1132739058 conn(0x7f044c06e000 legacy=0x7f044c0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 10 monclient: shutdown 2024-03-25T19:40:28.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:28.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 >> v1:172.21.15.178:6789/0 conn(0x7f04680b4d60 legacy=0x7f0468133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 shutdown_connections 2024-03-25T19:40:28.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 >> 172.21.15.110:0/2810557898 conn(0x7f04680617a0 msgr2=0x7f046813c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:28.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 shutdown_connections 2024-03-25T19:40:28.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.330+0000 7f046ec8d640 1 -- 172.21.15.110:0/2810557898 wait complete. 2024-03-25T19:40:28.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:28.331+0000 7f046ec8d640 1 librados: shutdown 2024-03-25T19:40:28.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:32.540 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:06.660 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:32.571 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:06.694 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:32.603 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:06.723 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:32.633 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:06.757 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:32.653 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:06.760 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:32.712 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:06.819 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:32.913 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:07.058 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:32.978 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:07.122 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.037 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:07.185 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.098 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:07.246 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.122 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:07.248 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.145 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-25T19:40:33.140+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:33.182 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:07.308 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.200 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-25T19:40:33.195+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:33.258 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:07.417 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.269 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:07.422 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.292 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-25T19:40:33.287+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:33.317 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:07.476 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.328 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:07.481 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-03-25T19:40:33.515+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:33.603 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:07.724 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.659 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:07.782 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-03-25T19:40:33.666+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:33.755 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-25T19:40:33.750+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:33.775 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:07.882 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.889 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-03-25T19:40:33.884+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:33.902 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-03-25T19:40:33.896+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:33.973 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:08.093 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:33.976 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:08.120 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.049 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:08.172 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.156 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:08.263 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.174 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:08.323 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.210 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:08.335 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.286 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:08.407 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.358 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:08.502 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.363 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:08.522 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.367 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:08.520 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.380 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:08.504 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.468 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:08.575 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.537 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:08.685 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.554 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:08.675 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.577 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:08.703 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.649 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:08.772 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.680 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:08.824 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.732 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:08.891 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.735 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:08.843 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.740 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:08.892 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.822 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:08.942 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.853 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.001 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.914 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.037 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.931 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.057 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.947 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.092 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:34.999 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.107 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.063 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.222 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.086 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.207 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.109 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.262 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.131 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.280 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.151 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:35.183 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.306 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.219 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.364 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.232 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.358 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.240+0000 7fd01d138640 10 monclient: get_monmap_and_config 2024-03-25T19:40:35.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.240+0000 7fd01d138640 10 monclient: build_initial_monmap 2024-03-25T19:40:35.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.240+0000 7fd01d138640 10 monclient: monmap: 2024-03-25T19:40:35.239 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:40:35.241155+0000 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:40:35.241155+0000 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:35.240 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:35.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.240+0000 7fd01d138640 1 Processor -- start 2024-03-25T19:40:35.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.240+0000 7fd01d138640 1 -- start start 2024-03-25T19:40:35.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.240+0000 7fd01d138640 10 monclient: init 2024-03-25T19:40:35.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:35.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:35.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 10 monclient(hunting): picked mon.noname-a con 0x7fd0180a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:35.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 10 monclient(hunting): picked mon.noname-b con 0x7fd01813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:35.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 10 monclient(hunting): picked mon.noname-c con 0x7fd0181430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:35.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd018142520 con 0x7fd0180a98f0 2024-03-25T19:40:35.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0180b28a0 con 0x7fd0181430c0 2024-03-25T19:40:35.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0180b2a30 con 0x7fd01813f710 2024-03-25T19:40:35.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:35.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd01d138640 10 monclient(hunting): authenticate will time out at 4206.382259s 2024-03-25T19:40:35.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd016d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd0180a98f0 0x7fd0180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:43464/0 (socket says 172.21.15.110:43464) 2024-03-25T19:40:35.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.241+0000 7fd016d9d640 1 -- 172.21.15.110:0/4047224275 learned_addr learned my addr 172.21.15.110:0/4047224275 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:35.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670694466 0 0) 0x7fd0180b28a0 con 0x7fd0181430c0 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd000003520 con 0x7fd0181430c0 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341466723 0 0) 0x7fd0180b2a30 con 0x7fd01813f710 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0180b28a0 con 0x7fd01813f710 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557866196 0 0) 0x7fd018142520 con 0x7fd0180a98f0 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 10 monclient(hunting): my global_id is 4197 2024-03-25T19:40:35.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.244+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0180b2a30 con 0x7fd0180a98f0 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004181779 0 0) 0x7fd0180b2a30 con 0x7fd0180a98f0 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 >> v1:172.21.15.110:6790/0 conn(0x7fd01813f710 legacy=0x7fd0181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 >> v1:172.21.15.178:6789/0 conn(0x7fd0181430c0 legacy=0x7fd01814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 1 monclient: found mon.noname-a 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0180b44c0 con 0x7fd0180a98f0 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 10 monclient: _finish_auth 0 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd0100015b0 con 0x7fd0180a98f0 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:35.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd015d9b640 10 monclient: dump: 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd01d138640 5 monclient: authenticate success, global_id 4197 2024-03-25T19:40:35.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.247+0000 7fd01d138640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:35.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.248+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd010001710 con 0x7fd0180a98f0 2024-03-25T19:40:35.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.248+0000 7fd015d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:35.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.248+0000 7fd015d9b640 1 -- 172.21.15.110:0/4047224275 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd010001b60 con 0x7fd0180a98f0 2024-03-25T19:40:35.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.248+0000 7fd015d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:35.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.248+0000 7fd015d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:35.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.248+0000 7fd015d9b640 10 monclient: dump: 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.248+0000 7fd01d138640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.248+0000 7fd01d138640 4 set_mon_vals no callback set 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.249+0000 7fd01d138640 10 monclient: shutdown 2024-03-25T19:40:35.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.249+0000 7fd01d138640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.249+0000 7fd01d138640 1 -- 172.21.15.110:0/4047224275 >> v1:172.21.15.110:6789/0 conn(0x7fd0180a98f0 legacy=0x7fd0180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.249+0000 7fd01d138640 1 -- 172.21.15.110:0/4047224275 shutdown_connections 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.249+0000 7fd01d138640 1 -- 172.21.15.110:0/4047224275 >> 172.21.15.110:0/4047224275 conn(0x7fd0180b7ea0 msgr2=0x7fd0180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.249+0000 7fd01d138640 1 -- 172.21.15.110:0/4047224275 shutdown_connections 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.249+0000 7fd01d138640 1 -- 172.21.15.110:0/4047224275 wait complete. 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 10 monclient: build_initial_monmap 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 10 monclient: monmap: 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:40:35.253 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 1 librados: starting msgr at 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 1 librados: starting objecter 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 1 Processor -- start 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 1 -- start start 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 1 librados: setting wanted keys 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 1 librados: calling monclient init 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.250+0000 7fd01d138640 10 monclient: init 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 10 monclient(hunting): picked mon.noname-c con 0x7fd0180a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 10 monclient(hunting): picked mon.noname-b con 0x7fd01813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:35.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 10 monclient(hunting): picked mon.noname-a con 0x7fd0181430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd018136f40 con 0x7fd0181430c0 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd018137220 con 0x7fd01813f710 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd018137500 con 0x7fd0180a98f0 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.251+0000 7fd01d138640 10 monclient(hunting): authenticate will time out at 4206.392350s 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.252+0000 7fd01659c640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fd01813f710 0x7fd0181d46d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:39318/0 (socket says 172.21.15.110:39318) 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.252+0000 7fd01659c640 1 -- 172.21.15.110:0/2185488767 learned_addr learned my addr 172.21.15.110:0/2185488767 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.252+0000 7fcfff7fe640 10 client.?.objecter ms_handle_connect 0x7fd01813f710 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.252+0000 7fcfff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.252+0000 7fcfff7fe640 10 client.?.objecter ms_handle_connect 0x7fd0180a98f0 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.252+0000 7fcfff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.252+0000 7fcfff7fe640 10 client.?.objecter ms_handle_connect 0x7fd0181430c0 2024-03-25T19:40:35.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.252+0000 7fcfff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:35.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200395392 0 0) 0x7fd018137220 con 0x7fd01813f710 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd004004eb0 con 0x7fd01813f710 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684396165 0 0) 0x7fd018137500 con 0x7fd0180a98f0 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd018137220 con 0x7fd0180a98f0 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093337254 0 0) 0x7fd018136f40 con 0x7fd0181430c0 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.258 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 10 monclient(hunting): my global_id is 4200 2024-03-25T19:40:35.258 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.254+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd018137500 con 0x7fd0181430c0 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 192589500 0 0) 0x7fd018137220 con 0x7fd0180a98f0 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd018136f40 con 0x7fd0180a98f0 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd010001a20 con 0x7fd0180a98f0 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:35.259 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2178065894 0 0) 0x7fd004004eb0 con 0x7fd01813f710 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd018137220 con 0x7fd01813f710 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd00c001d90 con 0x7fd01813f710 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.258+0000 7fcfff7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.260 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.259+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2702701630 0 0) 0x7fd018137500 con 0x7fd0181430c0 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.259+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd004004eb0 con 0x7fd0181430c0 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.259+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd0080016b0 con 0x7fd0181430c0 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.259+0000 7fcfff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.259+0000 7fcfff7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.259+0000 7fcfff7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.261 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:35.262 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:35.262 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:35.262 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:35.262 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:35.262 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:35.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.260+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1687712606 0 0) 0x7fd018137220 con 0x7fd01813f710 2024-03-25T19:40:35.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 >> v1:172.21.15.110:6790/0 conn(0x7fd0180a98f0 legacy=0x7fd0181daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 >> v1:172.21.15.110:6789/0 conn(0x7fd0181430c0 legacy=0x7fd0181e2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 1 monclient: found mon.b 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0181d41e0 con 0x7fd01813f710 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fcfff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 10 monclient: _renew_subs 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd018135960 con 0x7fd01813f710 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 10 monclient: _renew_subs 2024-03-25T19:40:35.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:35.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0181db830 con 0x7fd01813f710 2024-03-25T19:40:35.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.261+0000 7fd01d138640 1 librados: init done 2024-03-25T19:40:35.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.262+0000 7fd01d138640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:35.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.262+0000 7fd01d138640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:35.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.262+0000 7fd01d138640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:35.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.262+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0180627b0 con 0x7fd01813f710 2024-03-25T19:40:35.270 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.377 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.331 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.490 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.351 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.472 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.398 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.546 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.403 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.556 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.450 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.574 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.488 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.632 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.497+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd00c001fd0 con 0x7fd01813f710 2024-03-25T19:40:35.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.497+0000 7fcfff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:35.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.497+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd00c003de0 con 0x7fd01813f710 2024-03-25T19:40:35.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.497+0000 7fcfff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:35.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.497+0000 7fcfff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.497+0000 7fcfff7fe640 10 monclient: dump: 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:35.498 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.497+0000 7fd01559a640 4 set_mon_vals no callback set 2024-03-25T19:40:35.499 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.498+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fd00c004000 con 0x7fd01813f710 2024-03-25T19:40:35.499 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.498+0000 7fd016d9d640 10 monclient: get_auth_request con 0x7fd00406e160 auth_method 0 2024-03-25T19:40:35.499 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.499+0000 7fcfff7fe640 10 client.4213.objecter ms_handle_connect 0x7fd00406e160 2024-03-25T19:40:35.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.503+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fd00c054ba0 con 0x7fd01813f710 2024-03-25T19:40:35.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.503+0000 7fcfff7fe640 10 client.4213.objecter ms_dispatch 0x7fd018000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:35.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.503+0000 7fcfff7fe640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:35.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.503+0000 7fcfff7fe640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:35.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.503+0000 7fcfff7fe640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T19:40:35.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.503+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd00c0506b0 con 0x7fd01813f710 2024-03-25T19:40:35.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.503+0000 7fcfff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:35.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.503+0000 7fcfff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:35.520 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.646 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.541 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.649 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.596 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.754 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.616 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:09.737 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.667 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.815 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.677+0000 7fd01d138640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:35.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.678+0000 7fd01d138640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:35.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.678+0000 7fd01d138640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:40:35.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.678+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd018135c60 con 0x7fd01813f710 2024-03-25T19:40:35.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.694+0000 7fcfff7fe640 1 -- 172.21.15.110:0/2185488767 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fd00c0525e0 con 0x7fd01813f710 2024-03-25T19:40:35.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.694+0000 7fcfff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:35.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.694+0000 7fcfff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:35.694 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:40:35.694 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:35.694 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.847 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 10 librados: watch_flush enter 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 10 librados: watch_flush exit 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 10 client.4213.objecter successfully canceled tick 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 >> v1:172.21.15.178:6800/1132739058 conn(0x7fd00406e160 legacy=0x7fd004070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 10 monclient: shutdown 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 >> v1:172.21.15.178:6789/0 conn(0x7fd01813f710 legacy=0x7fd0181d46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 shutdown_connections 2024-03-25T19:40:35.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 >> 172.21.15.110:0/2185488767 conn(0x7fd0180b7ea0 msgr2=0x7fd018074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:35.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 shutdown_connections 2024-03-25T19:40:35.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 1 -- 172.21.15.110:0/2185488767 wait complete. 2024-03-25T19:40:35.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:35.696+0000 7fd01d138640 1 librados: shutdown 2024-03-25T19:40:35.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:35.716 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:09.840 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.758 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.902 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.808 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:09.916 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.813 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:09.939 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.860 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.019 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.881 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.001 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.939 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.087 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.980 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.104 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:35.982 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.135 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.023 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.167 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.072 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.180 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.105 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.231 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.125 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.284 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.203 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.351 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.273 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.426 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.300 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.445 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.351 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.472 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.393 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.519 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.397 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.556 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.405 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:10.526 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.470 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.618 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.514 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.638 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.569 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:10.692 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.572 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.725 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.591 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.698 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.645 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:10.752 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.669 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.827 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.683 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:10.809 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.797 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.941 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.851 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.995 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.865 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:11.018 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:36.954 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:11.102 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:37.008 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:11.156 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:37.152 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:11.310 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:37.206 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:11.365 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:37.249 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:11.375 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:37.314 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:11.440 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:37.399 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:11.552 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:37.455 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:11.607 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.349 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:16.472 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.390 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:16.510 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.392 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:16.500 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.511 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:42.567 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:16.693 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.642+0000 7fc434c4a640 10 monclient: get_monmap_and_config 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.642+0000 7fc434c4a640 10 monclient: build_initial_monmap 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.642+0000 7fc434c4a640 10 monclient: monmap: 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:40:42.643799+0000 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:40:42.643799+0000 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:42.642 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:42.643 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:42.643 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:42.643 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:42.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.643+0000 7fc434c4a640 1 Processor -- start 2024-03-25T19:40:42.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.643+0000 7fc434c4a640 1 -- start start 2024-03-25T19:40:42.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.643+0000 7fc434c4a640 10 monclient: init 2024-03-25T19:40:42.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:42.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fc4300b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fc4300b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fc43014be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4300af940 con 0x7fc4300b4d60 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4300b0a70 con 0x7fc4300b1730 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4300b0d50 con 0x7fc43014be90 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.644+0000 7fc434c4a640 10 monclient(hunting): authenticate will time out at 4213.785189s 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.645+0000 7fc42ed9d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fc43014be90 0x7fc43014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41716/0 (socket says 172.21.15.110:41716) 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.645+0000 7fc42ed9d640 1 -- 172.21.15.110:0/2535471016 learned_addr learned my addr 172.21.15.110:0/2535471016 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:42.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.645+0000 7fc42e59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc4300b4d60 0x7fc43013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58938/0 (socket says 172.21.15.110:58938) 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.647+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167189631 0 0) 0x7fc4300b0a70 con 0x7fc4300b1730 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc418003520 con 0x7fc4300b1730 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145888645 0 0) 0x7fc4300b0d50 con 0x7fc43014be90 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4300b0a70 con 0x7fc43014be90 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323589487 0 0) 0x7fc4300af940 con 0x7fc4300b4d60 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 10 monclient(hunting): my global_id is 4203 2024-03-25T19:40:42.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.648+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4300b0d50 con 0x7fc4300b4d60 2024-03-25T19:40:42.649 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:16.798 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 380917576 0 0) 0x7fc418003520 con 0x7fc4300b1730 2024-03-25T19:40:42.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 >> v1:172.21.15.110:6790/0 conn(0x7fc43014be90 legacy=0x7fc43014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 >> v1:172.21.15.110:6789/0 conn(0x7fc4300b4d60 legacy=0x7fc43013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 1 monclient: found mon.noname-c 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4300b1030 con 0x7fc4300b1730 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 10 monclient: _finish_auth 0 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc424001690 con 0x7fc4300b1730 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc434c4a640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc434c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc42d59a640 10 monclient: dump: 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:42.652 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.653 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.653 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.653 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:42.653 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:42.653 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:42.653 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:42.653 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:42.653 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.651+0000 7fc434c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:42.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.653+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4240018b0 con 0x7fc4300b1730 2024-03-25T19:40:42.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.653+0000 7fc42d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:42.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.653+0000 7fc42d59a640 1 -- 172.21.15.110:0/2535471016 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc424001c40 con 0x7fc4300b1730 2024-03-25T19:40:42.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.653+0000 7fc434c4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:42.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.653+0000 7fc434c4a640 4 set_mon_vals no callback set 2024-03-25T19:40:42.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.653+0000 7fc42d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.653+0000 7fc42d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.653+0000 7fc42d59a640 10 monclient: dump: 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:42.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.654+0000 7fc434c4a640 10 monclient: shutdown 2024-03-25T19:40:42.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.654+0000 7fc434c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:42.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.654+0000 7fc434c4a640 1 -- 172.21.15.110:0/2535471016 >> v1:172.21.15.178:6789/0 conn(0x7fc4300b1730 legacy=0x7fc4300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.654+0000 7fc434c4a640 1 -- 172.21.15.110:0/2535471016 shutdown_connections 2024-03-25T19:40:42.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.654+0000 7fc434c4a640 1 -- 172.21.15.110:0/2535471016 >> 172.21.15.110:0/2535471016 conn(0x7fc4300617a0 msgr2=0x7fc430061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:42.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.654+0000 7fc434c4a640 1 -- 172.21.15.110:0/2535471016 shutdown_connections 2024-03-25T19:40:42.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.655+0000 7fc434c4a640 1 -- 172.21.15.110:0/2535471016 wait complete. 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 10 monclient: build_initial_monmap 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 10 monclient: monmap: 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:40:42.659 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 1 librados: starting msgr at 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 1 librados: starting objecter 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 1 Processor -- start 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 1 -- start start 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 1 librados: setting wanted keys 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 1 librados: calling monclient init 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.656+0000 7fc434c4a640 10 monclient: init 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fc4300b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fc4300b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:42.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fc43014be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43006e310 con 0x7fc4300b1730 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43006de80 con 0x7fc4300b4d60 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4301d7e00 con 0x7fc43014be90 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.657+0000 7fc434c4a640 10 monclient(hunting): authenticate will time out at 4213.798816s 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.658+0000 7fc42e59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc4300b1730 0x7fc43006f730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58946/0 (socket says 172.21.15.110:58946) 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.658+0000 7fc42e59c640 1 -- 172.21.15.110:0/3493458574 learned_addr learned my addr 172.21.15.110:0/3493458574 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.658+0000 7fc41effd640 10 client.?.objecter ms_handle_connect 0x7fc4300b4d60 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.658+0000 7fc41effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.659+0000 7fc41effd640 10 client.?.objecter ms_handle_connect 0x7fc4300b1730 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.659+0000 7fc41effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.659+0000 7fc41effd640 10 client.?.objecter ms_handle_connect 0x7fc43014be90 2024-03-25T19:40:42.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.659+0000 7fc41effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337054061 0 0) 0x7fc43006de80 con 0x7fc4300b4d60 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc410004eb0 con 0x7fc4300b4d60 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847435533 0 0) 0x7fc4300b0d50 con 0x7fc4300b1730 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43006de80 con 0x7fc4300b1730 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512287526 0 0) 0x7fc4301d7e00 con 0x7fc43014be90 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 10 monclient(hunting): my global_id is 4223 2024-03-25T19:40:42.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.661+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc410005360 con 0x7fc43014be90 2024-03-25T19:40:42.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.665+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3240544529 0 0) 0x7fc410004eb0 con 0x7fc4300b4d60 2024-03-25T19:40:42.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.665+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4301d7e00 con 0x7fc4300b4d60 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.665+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc4240015d0 con 0x7fc4300b4d60 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.665+0000 7fc41effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.665+0000 7fc41effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.665+0000 7fc41effd640 10 monclient(hunting): dump: 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:42.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.665+0000 7fc41effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.666+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3447231467 0 0) 0x7fc43006de80 con 0x7fc4300b1730 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.666+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc410004eb0 con 0x7fc4300b1730 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.666+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc420001700 con 0x7fc4300b1730 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.666+0000 7fc41effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.666+0000 7fc41effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.666+0000 7fc41effd640 10 monclient(hunting): dump: 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:42.669 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:42.670 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:42.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1286409122 0 0) 0x7fc410004eb0 con 0x7fc4300b1730 2024-03-25T19:40:42.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 >> v1:172.21.15.110:6790/0 conn(0x7fc43014be90 legacy=0x7fc4301d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 >> v1:172.21.15.178:6789/0 conn(0x7fc4300b4d60 legacy=0x7fc4301d6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 1 monclient: found mon.a 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43006f030 con 0x7fc4300b1730 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 10 monclient: _finish_auth 0 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc41effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 5 monclient: authenticate success, global_id 4206 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 10 monclient: _renew_subs 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43006f310 con 0x7fc4300b1730 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 10 client.4206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 10 monclient: _renew_subs 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:42.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc430134800 con 0x7fc4300b1730 2024-03-25T19:40:42.674 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.669+0000 7fc434c4a640 1 librados: init done 2024-03-25T19:40:42.674 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.670+0000 7fc434c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.674 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.670+0000 7fc434c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.674 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.670+0000 7fc434c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:42.674 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.670+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430142320 con 0x7fc4300b1730 2024-03-25T19:40:42.707 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:16.866 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.814 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:16.958 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.876 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:17.029 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:42.885 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.886+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc420001c10 con 0x7fc4300b1730 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.886+0000 7fc41effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.886+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc420001700 con 0x7fc4300b1730 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.886+0000 7fc41effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.886+0000 7fc41effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.886+0000 7fc41effd640 10 monclient: dump: 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.886 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:42.887 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.887 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:42.887 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:42.887 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:42.887 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:42.887 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:42.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.886+0000 7fc42cd99640 4 set_mon_vals no callback set 2024-03-25T19:40:42.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.920+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fc42001a650 con 0x7fc4300b1730 2024-03-25T19:40:42.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.921+0000 7fc42dd9b640 10 monclient: get_auth_request con 0x7fc41006e520 auth_method 0 2024-03-25T19:40:42.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.922+0000 7fc41effd640 10 client.4206.objecter ms_handle_connect 0x7fc41006e520 2024-03-25T19:40:42.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.927+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fc420054100 con 0x7fc4300b1730 2024-03-25T19:40:42.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.927+0000 7fc41effd640 10 client.4206.objecter ms_dispatch 0x7fc430000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:42.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.927+0000 7fc41effd640 3 client.4206.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:42.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.927+0000 7fc41effd640 3 client.4206.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:42.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.927+0000 7fc41effd640 20 client.4206.objecter dump_active .. 0 homeless 2024-03-25T19:40:42.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.927+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc42004fc10 con 0x7fc4300b1730 2024-03-25T19:40:42.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.927+0000 7fc41effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:42.927+0000 7fc41effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:43.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.187+0000 7fc434c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:43.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.187+0000 7fc434c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:43.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.187+0000 7fc434c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:43.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.187+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc43006e310 con 0x7fc4300b1730 2024-03-25T19:40:43.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.210+0000 7fc41effd640 1 -- 172.21.15.110:0/3493458574 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fc43006e310 con 0x7fc4300b1730 2024-03-25T19:40:43.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.210+0000 7fc41effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:43.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.210+0000 7fc41effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:43.210 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:40:43.210 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 10 librados: watch_flush enter 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 10 librados: watch_flush exit 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 20 client.4206.objecter shutdown clearing up homeless session... 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 10 client.4206.objecter successfully canceled tick 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 >> v1:172.21.15.178:6800/1132739058 conn(0x7fc41006e520 legacy=0x7fc4100709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 10 monclient: shutdown 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 >> v1:172.21.15.110:6789/0 conn(0x7fc4300b1730 legacy=0x7fc43006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 shutdown_connections 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 >> 172.21.15.110:0/3493458574 conn(0x7fc4300617a0 msgr2=0x7fc43013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:43.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 shutdown_connections 2024-03-25T19:40:43.214 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 1 -- 172.21.15.110:0/3493458574 wait complete. 2024-03-25T19:40:43.214 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:43.213+0000 7fc434c4a640 1 librados: shutdown 2024-03-25T19:40:43.226 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:47.535 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:21.658 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:47.538 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:21.662 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:47.597 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:21.705 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:47.601 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:21.709 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:47.622 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:21.743 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:47.628 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:21.749 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:47.820 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:21.946 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:47.826 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:21.952 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:48.404 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:22.552 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:48.408 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:22.556 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:48.500 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:22.659 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:48.504 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:22.663 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:48.618 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:22.771 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:48.622 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:22.775 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:48.793 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:22.937 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:48.796 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:22.941 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:50.033 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:50.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.123+0000 7f28c7625640 10 monclient: get_monmap_and_config 2024-03-25T19:40:50.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.123+0000 7f28c7625640 10 monclient: build_initial_monmap 2024-03-25T19:40:50.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.123+0000 7f28c7625640 10 monclient: monmap: 2024-03-25T19:40:50.124 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:50.124 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:40:50.124064+0000 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:40:50.124064+0000 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.123+0000 7f28c7625640 1 Processor -- start 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.123+0000 7f28c7625640 1 -- start start 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.123+0000 7f28c7625640 10 monclient: init 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:50.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 10 monclient(hunting): picked mon.noname-a con 0x7f28c00b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 10 monclient(hunting): picked mon.noname-c con 0x7f28c00b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 10 monclient(hunting): picked mon.noname-b con 0x7f28c014be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c00af940 con 0x7f28c00b4d60 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c00b0a70 con 0x7f28c00b1730 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c00b0d50 con 0x7f28c014be90 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.124+0000 7f28c7625640 10 monclient(hunting): authenticate will time out at 4221.265416s 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.125+0000 7f28c5bc2640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f28c014be90 0x7f28c014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:41732/0 (socket says 172.21.15.110:41732) 2024-03-25T19:40:50.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.125+0000 7f28c5bc2640 1 -- 172.21.15.110:0/3174532075 learned_addr learned my addr 172.21.15.110:0/3174532075 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:50.128 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964044720 0 0) 0x7f28c00b0d50 con 0x7f28c014be90 2024-03-25T19:40:50.128 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 10 monclient(hunting): my global_id is 4226 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ac003520 con 0x7f28c014be90 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026408002 0 0) 0x7f28c00af940 con 0x7f28c00b4d60 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.127+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c00b0d50 con 0x7f28c00b4d60 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.128+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174665542 0 0) 0x7f28c00b0a70 con 0x7f28c00b1730 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.128+0000 7f28b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.128+0000 7f28b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.128+0000 7f28b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.128+0000 7f28b7fff640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:40:50.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.128+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c00af940 con 0x7f28c00b1730 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.130+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820064325 0 0) 0x7f28ac003520 con 0x7f28c014be90 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.130+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 >> v1:172.21.15.178:6789/0 conn(0x7f28c00b1730 legacy=0x7f28c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.130+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 >> v1:172.21.15.110:6789/0 conn(0x7f28c00b4d60 legacy=0x7f28c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.130+0000 7f28b7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.130+0000 7f28b7fff640 1 monclient: found mon.noname-b 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c00b1030 con 0x7f28c014be90 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:50.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f28b8001a10 con 0x7f28c014be90 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28b7fff640 10 monclient: dump: 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:50.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28c7625640 5 monclient: authenticate success, global_id 4226 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.131+0000 7f28c7625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.132+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28b8001b70 con 0x7f28c014be90 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.132+0000 7f28b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.132+0000 7f28b7fff640 1 -- 172.21.15.110:0/3174532075 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f28b8001f80 con 0x7f28c014be90 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.132+0000 7f28b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.132+0000 7f28b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.132+0000 7f28b7fff640 10 monclient: dump: 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:50.138 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.132+0000 7f28c7625640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.132+0000 7f28c7625640 4 set_mon_vals no callback set 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.133+0000 7f28c7625640 10 monclient: shutdown 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.133+0000 7f28c7625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.133+0000 7f28c7625640 1 -- 172.21.15.110:0/3174532075 >> v1:172.21.15.110:6790/0 conn(0x7f28c014be90 legacy=0x7f28c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.133+0000 7f28c7625640 1 -- 172.21.15.110:0/3174532075 shutdown_connections 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.133+0000 7f28c7625640 1 -- 172.21.15.110:0/3174532075 >> 172.21.15.110:0/3174532075 conn(0x7f28c00617a0 msgr2=0x7f28c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.133+0000 7f28c7625640 1 -- 172.21.15.110:0/3174532075 shutdown_connections 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.133+0000 7f28c7625640 1 -- 172.21.15.110:0/3174532075 wait complete. 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 10 monclient: build_initial_monmap 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 10 monclient: monmap: 2024-03-25T19:40:50.139 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 1 librados: starting msgr at 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 1 librados: starting objecter 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 1 Processor -- start 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 1 -- start start 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 1 librados: setting wanted keys 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 1 librados: calling monclient init 2024-03-25T19:40:50.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.134+0000 7f28c7625640 10 monclient: init 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 10 monclient(hunting): picked mon.noname-c con 0x7f28c00b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 10 monclient(hunting): picked mon.noname-b con 0x7f28c00b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 10 monclient(hunting): picked mon.noname-a con 0x7f28c014be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c01364f0 con 0x7f28c014be90 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c0137040 con 0x7f28c00b4d60 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c0137320 con 0x7f28c00b1730 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.135+0000 7f28c7625640 10 monclient(hunting): authenticate will time out at 4221.276508s 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.136+0000 7f28c5bc2640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f28c014be90 0x7f28c01e3440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58960/0 (socket says 172.21.15.110:58960) 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.136+0000 7f28c5bc2640 1 -- 172.21.15.110:0/700983980 learned_addr learned my addr 172.21.15.110:0/700983980 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.136+0000 7f28b5ffb640 10 client.?.objecter ms_handle_connect 0x7f28c00b1730 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.136+0000 7f28b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:50.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.136+0000 7f28b5ffb640 10 client.?.objecter ms_handle_connect 0x7f28c00b4d60 2024-03-25T19:40:50.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.136+0000 7f28b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:50.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.136+0000 7f28b5ffb640 10 client.?.objecter ms_handle_connect 0x7f28c014be90 2024-03-25T19:40:50.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.136+0000 7f28b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:50.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.138+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606126801 0 0) 0x7f28c0137320 con 0x7f28c00b1730 2024-03-25T19:40:50.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): my global_id is 4229 2024-03-25T19:40:50.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a4004eb0 con 0x7f28c00b1730 2024-03-25T19:40:50.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682427808 0 0) 0x7f28c01364f0 con 0x7f28c014be90 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c0137320 con 0x7f28c014be90 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098457137 0 0) 0x7f28c0137040 con 0x7f28c00b4d60 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.139+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c01364f0 con 0x7f28c00b4d60 2024-03-25T19:40:50.146 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.143+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3201426972 0 0) 0x7f28a4004eb0 con 0x7f28c00b1730 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.143+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28c0137040 con 0x7f28c00b1730 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.143+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f28bc001970 con 0x7f28c00b1730 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.143+0000 7f28b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.143+0000 7f28b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.143+0000 7f28b5ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:50.147 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.143+0000 7f28b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.144+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2351830711 0 0) 0x7f28c0137320 con 0x7f28c014be90 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.144+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28a4004eb0 con 0x7f28c014be90 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.144+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f28b8001e80 con 0x7f28c014be90 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.144+0000 7f28b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.144+0000 7f28b5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.144+0000 7f28b5ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:50.148 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.145+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375362588 0 0) 0x7f28c01364f0 con 0x7f28c00b4d60 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.145+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28c0137320 con 0x7f28c00b4d60 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.145+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f28b00018f0 con 0x7f28c00b4d60 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.145+0000 7f28b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.145+0000 7f28b5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.145+0000 7f28b5ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.149 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:50.150 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:50.150 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:50.150 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:50.150 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.146+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3686330808 0 0) 0x7f28c0137040 con 0x7f28c00b1730 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 >> v1:172.21.15.178:6789/0 conn(0x7f28c00b4d60 legacy=0x7f28c012ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 >> v1:172.21.15.110:6789/0 conn(0x7f28c014be90 legacy=0x7f28c01e3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 1 monclient: found mon.c 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c013e1f0 con 0x7f28c00b1730 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:50.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 5 monclient: authenticate success, global_id 4229 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 10 monclient: _renew_subs 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c013e3e0 con 0x7f28c00b1730 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 10 monclient: _renew_subs 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c012f5d0 con 0x7f28c00b1730 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.147+0000 7f28c7625640 1 librados: init done 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.148+0000 7f28c7625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.148+0000 7f28c7625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.148+0000 7f28c7625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:50.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.148+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c00b2270 con 0x7f28c00b1730 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.164+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28bc0035f0 con 0x7f28c00b1730 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.164+0000 7f28b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.164+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f28bc003900 con 0x7f28c00b1730 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.164+0000 7f28b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.164+0000 7f28b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.164+0000 7f28b5ffb640 10 monclient: dump: 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.165 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:50.166 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:50.166 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:50.166 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:50.166 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:50.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.164+0000 7f28b77fe640 4 set_mon_vals no callback set 2024-03-25T19:40:50.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.165+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f28bc019910 con 0x7f28c00b1730 2024-03-25T19:40:50.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.165+0000 7f28c4bc0640 10 monclient: get_auth_request con 0x7f28a406e040 auth_method 0 2024-03-25T19:40:50.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.166+0000 7f28b5ffb640 10 client.4229.objecter ms_handle_connect 0x7f28a406e040 2024-03-25T19:40:50.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.519+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f28bc0535d0 con 0x7f28c00b1730 2024-03-25T19:40:50.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.519+0000 7f28b5ffb640 10 client.4229.objecter ms_dispatch 0x7f28c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:50.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.519+0000 7f28b5ffb640 3 client.4229.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:50.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.519+0000 7f28b5ffb640 3 client.4229.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:50.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.519+0000 7f28b5ffb640 20 client.4229.objecter dump_active .. 0 homeless 2024-03-25T19:40:50.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.519+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28bc04ef10 con 0x7f28c00b1730 2024-03-25T19:40:50.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.519+0000 7f28b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:50.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.519+0000 7f28b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:50.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.794+0000 7f28c7625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:50.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.794+0000 7f28c7625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:50.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.794+0000 7f28c7625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:40:50.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.794+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28c00627b0 con 0x7f28c00b1730 2024-03-25T19:40:50.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.867+0000 7f28b5ffb640 1 -- 172.21.15.110:0/700983980 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f28bc051010 con 0x7f28c00b1730 2024-03-25T19:40:50.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.867+0000 7f28b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:50.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.867+0000 7f28b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:50.867 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:40:50.867 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:50.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.869+0000 7f28c7625640 10 librados: watch_flush enter 2024-03-25T19:40:50.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.869+0000 7f28c7625640 10 librados: watch_flush exit 2024-03-25T19:40:50.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.869+0000 7f28c7625640 20 client.4229.objecter shutdown clearing up homeless session... 2024-03-25T19:40:50.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.869+0000 7f28c7625640 10 client.4229.objecter successfully canceled tick 2024-03-25T19:40:50.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.870+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 >> v1:172.21.15.178:6800/1132739058 conn(0x7f28a406e040 legacy=0x7f28a4070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.870+0000 7f28c7625640 10 monclient: shutdown 2024-03-25T19:40:50.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.870+0000 7f28c7625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:50.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.870+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 >> v1:172.21.15.110:6790/0 conn(0x7f28c00b1730 legacy=0x7f28c0135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.870+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 shutdown_connections 2024-03-25T19:40:50.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.870+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 >> 172.21.15.110:0/700983980 conn(0x7f28c00617a0 msgr2=0x7f28c013c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:50.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.871+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 shutdown_connections 2024-03-25T19:40:50.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.871+0000 7f28c7625640 1 -- 172.21.15.110:0/700983980 wait complete. 2024-03-25T19:40:50.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:50.872+0000 7f28c7625640 1 librados: shutdown 2024-03-25T19:40:50.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:51.727 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:00:25.850 35311== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.017 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:00:26.138 35313== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.096 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:00:26.203 35329== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.405 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:00:26.530 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.916 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:27.074 35559== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.283 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:27.432 35562== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.351 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:27.504 35560== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.567 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:27.711 35564== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:57.689 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:57.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.801+0000 7f427044e640 10 monclient: get_monmap_and_config 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.801+0000 7f427044e640 10 monclient: build_initial_monmap 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.801+0000 7f427044e640 10 monclient: monmap: 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:40:57.802292+0000 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:40:57.802292+0000 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:40:57.801 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:57.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.802+0000 7f427044e640 1 Processor -- start 2024-03-25T19:40:57.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.802+0000 7f427044e640 1 -- start start 2024-03-25T19:40:57.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.802+0000 7f427044e640 10 monclient: init 2024-03-25T19:40:57.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.803+0000 7f427044e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:57.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.803+0000 7f427044e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.803+0000 7f427044e640 10 monclient(hunting): picked mon.noname-a con 0x7f42680a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.803+0000 7f427044e640 10 monclient(hunting): picked mon.noname-c con 0x7f426813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.803+0000 7f427044e640 10 monclient(hunting): picked mon.noname-b con 0x7f42681430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.804+0000 7f427044e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4268142520 con 0x7f42680a98f0 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.804+0000 7f427044e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42680b28a0 con 0x7f426813f710 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.804+0000 7f427044e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42680b2a30 con 0x7f42681430c0 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.804+0000 7f427044e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.804+0000 7f427044e640 10 monclient(hunting): authenticate will time out at 4228.945033s 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.804+0000 7f426d9e9640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f426813f710 0x7f42681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:52384/0 (socket says 172.21.15.110:52384) 2024-03-25T19:40:57.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.804+0000 7f426d9e9640 1 -- 172.21.15.110:0/2007533889 learned_addr learned my addr 172.21.15.110:0/2007533889 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:57.814 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.814+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713227645 0 0) 0x7f4268142520 con 0x7f42680a98f0 2024-03-25T19:40:57.814 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.814+0000 7f426d1e8640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.814 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.814+0000 7f426d1e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.814 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.814+0000 7f426d1e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.814 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.814+0000 7f426d1e8640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:40:57.814 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.815+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4254003520 con 0x7f42680a98f0 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 995843504 0 0) 0x7f4254003520 con 0x7f42680a98f0 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 >> v1:172.21.15.110:6790/0 conn(0x7f42681430c0 legacy=0x7f426814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 >> v1:172.21.15.178:6789/0 conn(0x7f426813f710 legacy=0x7f42681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 10 monclient: _finish_hunting 0 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 1 monclient: found mon.noname-a 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42680b44c0 con 0x7f42680a98f0 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 10 monclient: _finish_auth 0 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 10 monclient: _check_auth_tickets 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4264001690 con 0x7f42680a98f0 2024-03-25T19:40:57.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.820+0000 7f426d1e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.821+0000 7f426d1e8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.821+0000 7f426d1e8640 10 monclient: dump: 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.821+0000 7f427044e640 5 monclient: authenticate success, global_id 4215 2024-03-25T19:40:57.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.821+0000 7f427044e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:57.826 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42640018b0 con 0x7f42680a98f0 2024-03-25T19:40:57.826 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f426d1e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:57.826 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f426d1e8640 1 -- 172.21.15.110:0/2007533889 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4264001c40 con 0x7f42680a98f0 2024-03-25T19:40:57.826 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f426d1e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.826 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f426d1e8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f426d1e8640 10 monclient: dump: 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f427044e640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f427044e640 4 set_mon_vals no callback set 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f427044e640 10 monclient: shutdown 2024-03-25T19:40:57.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f427044e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:57.828 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.826+0000 7f427044e640 1 -- 172.21.15.110:0/2007533889 >> v1:172.21.15.110:6789/0 conn(0x7f42680a98f0 legacy=0x7f42680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.828 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.827+0000 7f427044e640 1 -- 172.21.15.110:0/2007533889 shutdown_connections 2024-03-25T19:40:57.828 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.827+0000 7f427044e640 1 -- 172.21.15.110:0/2007533889 >> 172.21.15.110:0/2007533889 conn(0x7f42680b7ea0 msgr2=0x7f42680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:57.828 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.827+0000 7f427044e640 1 -- 172.21.15.110:0/2007533889 shutdown_connections 2024-03-25T19:40:57.828 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.827+0000 7f427044e640 1 -- 172.21.15.110:0/2007533889 wait complete. 2024-03-25T19:40:57.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 10 monclient: build_initial_monmap 2024-03-25T19:40:57.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 10 monclient: monmap: 2024-03-25T19:40:57.830 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 1 librados: starting msgr at 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 1 librados: starting objecter 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 1 Processor -- start 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 1 -- start start 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 1 librados: setting wanted keys 2024-03-25T19:40:57.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 1 librados: calling monclient init 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.828+0000 7f427044e640 10 monclient: init 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 10 monclient(hunting): picked mon.noname-c con 0x7f42680a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 10 monclient(hunting): picked mon.noname-b con 0x7f426813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 10 monclient(hunting): picked mon.noname-a con 0x7f42681430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4268138c70 con 0x7f42681430c0 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42681367d0 con 0x7f426813f710 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4268136960 con 0x7f42680a98f0 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.829+0000 7f427044e640 10 monclient(hunting): authenticate will time out at 4228.970774s 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.830+0000 7f426e9eb640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f42681430c0 0x7f42681e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58982/0 (socket says 172.21.15.110:58982) 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.830+0000 7f426e9eb640 1 -- 172.21.15.110:0/986210080 learned_addr learned my addr 172.21.15.110:0/986210080 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.831+0000 7f425effd640 10 client.?.objecter ms_handle_connect 0x7f42681430c0 2024-03-25T19:40:57.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.831+0000 7f425effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:57.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.833+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537969056 0 0) 0x7f4268138c70 con 0x7f42681430c0 2024-03-25T19:40:57.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.833+0000 7f425effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.833+0000 7f425effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.833+0000 7f425effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.833+0000 7f425effd640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:40:57.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.833+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f424c004eb0 con 0x7f42681430c0 2024-03-25T19:40:57.838 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.839+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272558771 0 0) 0x7f424c004eb0 con 0x7f42681430c0 2024-03-25T19:40:57.838 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.839+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4268138c70 con 0x7f42681430c0 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.839+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4260001c30 con 0x7f42681430c0 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.839+0000 7f425effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.839+0000 7f425effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.839+0000 7f425effd640 10 monclient(hunting): dump: 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:57.839 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:57.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.839+0000 7f425effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:57.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2012162887 0 0) 0x7f4268138c70 con 0x7f42681430c0 2024-03-25T19:40:57.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 >> v1:172.21.15.110:6790/0 conn(0x7f42680a98f0 legacy=0x7f4268137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 >> v1:172.21.15.178:6789/0 conn(0x7f426813f710 legacy=0x7f4268132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:57.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 1 monclient: found mon.a 2024-03-25T19:40:57.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:57.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:57.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4268139630 con 0x7f42681430c0 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 10 monclient: _finish_auth 0 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f425effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f427044e640 5 monclient: authenticate success, global_id 4218 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f427044e640 10 monclient: _renew_subs 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f427044e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.844+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4268139880 con 0x7f42681430c0 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 10 monclient: _renew_subs 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42681337f0 con 0x7f42681430c0 2024-03-25T19:40:57.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 1 librados: init done 2024-03-25T19:40:57.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:57.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:57.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:57.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.845+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42680627b0 con 0x7f42681430c0 2024-03-25T19:40:57.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.848+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4260001d90 con 0x7f42681430c0 2024-03-25T19:40:57.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.848+0000 7f425effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:57.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.848+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4260003bb0 con 0x7f42681430c0 2024-03-25T19:40:57.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.848+0000 7f425effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.848+0000 7f425effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:57.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.848+0000 7f425effd640 10 monclient: dump: 2024-03-25T19:40:57.847 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:40:57.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:57.848+0000 7f426c9e7640 4 set_mon_vals no callback set 2024-03-25T19:40:58.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.203+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f4260003d10 con 0x7f42681430c0 2024-03-25T19:40:58.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.204+0000 7f426e1ea640 10 monclient: get_auth_request con 0x7f424c06dbc0 auth_method 0 2024-03-25T19:40:58.204 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.205+0000 7f425effd640 10 client.4218.objecter ms_handle_connect 0x7f424c06dbc0 2024-03-25T19:40:58.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.208+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f4260054b00 con 0x7f42681430c0 2024-03-25T19:40:58.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.208+0000 7f425effd640 10 client.4218.objecter ms_dispatch 0x7f4268000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:58.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.208+0000 7f425effd640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:58.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.208+0000 7f425effd640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:58.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.208+0000 7f425effd640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T19:40:58.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.208+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4260050440 con 0x7f42681430c0 2024-03-25T19:40:58.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.208+0000 7f425effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:58.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.208+0000 7f425effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:58.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.391+0000 7f427044e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:58.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.392+0000 7f427044e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:58.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.392+0000 7f427044e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:40:58.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.392+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4268069f40 con 0x7f42681430c0 2024-03-25T19:40:58.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.416+0000 7f425effd640 1 -- 172.21.15.110:0/986210080 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f4260052540 con 0x7f42681430c0 2024-03-25T19:40:58.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.416+0000 7f425effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:58.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.416+0000 7f425effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:58.415 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:40:58.416 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.418+0000 7f427044e640 10 librados: watch_flush enter 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.418+0000 7f427044e640 10 librados: watch_flush exit 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.418+0000 7f427044e640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.418+0000 7f427044e640 10 client.4218.objecter successfully canceled tick 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 >> v1:172.21.15.178:6800/1132739058 conn(0x7f424c06dbc0 legacy=0x7f424c070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 10 monclient: shutdown 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 >> v1:172.21.15.110:6789/0 conn(0x7f42681430c0 legacy=0x7f42681e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 shutdown_connections 2024-03-25T19:40:58.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 >> 172.21.15.110:0/986210080 conn(0x7f42680b7ea0 msgr2=0x7f4268074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:58.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 shutdown_connections 2024-03-25T19:40:58.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 1 -- 172.21.15.110:0/986210080 wait complete. 2024-03-25T19:40:58.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:40:58.419+0000 7f427044e640 1 librados: shutdown 2024-03-25T19:40:58.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:00.255 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-25T19:41:00.245+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:41:00.555 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-25T19:41:00.553+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:41:01.090 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-25T19:41:01.081+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:41:01.270 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-25T19:41:01.268+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:41:01.859 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-03-25T19:41:01.855+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:41:02.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-03-25T19:41:02.667+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:41:02.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-03-25T19:41:02.675+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:41:03.071 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-03-25T19:41:03.067+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:41:05.235 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.350+0000 7f96f4081640 10 monclient: get_monmap_and_config 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.350+0000 7f96f4081640 10 monclient: build_initial_monmap 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.350+0000 7f96f4081640 10 monclient: monmap: 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:41:05.351358+0000 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:41:05.351358+0000 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:05.350 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:05.351 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:41:05.351 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:41:05.351 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:05.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 1 Processor -- start 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 1 -- start start 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 10 monclient: init 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 10 monclient(hunting): picked mon.noname-a con 0x7f96ec0a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 10 monclient(hunting): picked mon.noname-c con 0x7f96ec13f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 10 monclient(hunting): picked mon.noname-b con 0x7f96ec1430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec142520 con 0x7f96ec0a98f0 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0b28a0 con 0x7f96ec13f710 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0b2a30 con 0x7f96ec1430c0 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:05.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.351+0000 7f96f4081640 10 monclient(hunting): authenticate will time out at 4236.492787s 2024-03-25T19:41:05.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.359+0000 7f96f1e1d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f96ec0a98f0 0x7f96ec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:57634/0 (socket says 172.21.15.110:57634) 2024-03-25T19:41:05.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.359+0000 7f96f1e1d640 1 -- 172.21.15.110:0/882119332 learned_addr learned my addr 172.21.15.110:0/882119332 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:05.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.376+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650224509 0 0) 0x7f96ec142520 con 0x7f96ec0a98f0 2024-03-25T19:41:05.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.376+0000 7f96f0e1b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:05.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.376+0000 7f96f0e1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:05.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.376+0000 7f96f0e1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:05.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.376+0000 7f96f0e1b640 10 monclient(hunting): my global_id is 4245 2024-03-25T19:41:05.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.376+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d8003520 con 0x7f96ec0a98f0 2024-03-25T19:41:05.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.378+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274206698 0 0) 0x7f96ec0b28a0 con 0x7f96ec13f710 2024-03-25T19:41:05.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.378+0000 7f96f0e1b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:05.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.378+0000 7f96f0e1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:05.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.378+0000 7f96f0e1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:05.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.378+0000 7f96f0e1b640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:41:05.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.378+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec142520 con 0x7f96ec13f710 2024-03-25T19:41:05.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.383+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281159527 0 0) 0x7f96ec0b2a30 con 0x7f96ec1430c0 2024-03-25T19:41:05.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.383+0000 7f96f0e1b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:05.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.383+0000 7f96f0e1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:05.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.383+0000 7f96f0e1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:05.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.383+0000 7f96f0e1b640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:41:05.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.383+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec0b28a0 con 0x7f96ec1430c0 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.388+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568355532 0 0) 0x7f96ec142520 con 0x7f96ec13f710 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.388+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 >> v1:172.21.15.110:6790/0 conn(0x7f96ec1430c0 legacy=0x7f96ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.388+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 >> v1:172.21.15.110:6789/0 conn(0x7f96ec0a98f0 legacy=0x7f96ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 1 monclient: found mon.noname-c 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec0b44c0 con 0x7f96ec13f710 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 10 monclient: _finish_auth 0 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:05.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f96dc001670 con 0x7f96ec13f710 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f4081640 5 monclient: authenticate success, global_id 4255 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f4081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f0e1b640 10 monclient: dump: 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:05.390 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:05.391 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:05.391 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:05.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.389+0000 7f96f4081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:05.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.403+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96dc0017d0 con 0x7f96ec13f710 2024-03-25T19:41:05.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.403+0000 7f96f0e1b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:05.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.403+0000 7f96f0e1b640 1 -- 172.21.15.110:0/882119332 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f96dc001be0 con 0x7f96ec13f710 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.403+0000 7f96f0e1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.403+0000 7f96f0e1b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.403+0000 7f96f0e1b640 10 monclient: dump: 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.403+0000 7f96f4081640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:05.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.403+0000 7f96f4081640 4 set_mon_vals no callback set 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.404+0000 7f96f4081640 10 monclient: shutdown 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.404+0000 7f96f4081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.404+0000 7f96f4081640 1 -- 172.21.15.110:0/882119332 >> v1:172.21.15.178:6789/0 conn(0x7f96ec13f710 legacy=0x7f96ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.404+0000 7f96f4081640 1 -- 172.21.15.110:0/882119332 shutdown_connections 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.404+0000 7f96f4081640 1 -- 172.21.15.110:0/882119332 >> 172.21.15.110:0/882119332 conn(0x7f96ec0b7ea0 msgr2=0x7f96ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.404+0000 7f96f4081640 1 -- 172.21.15.110:0/882119332 shutdown_connections 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.404+0000 7f96f4081640 1 -- 172.21.15.110:0/882119332 wait complete. 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 10 monclient: build_initial_monmap 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 10 monclient: monmap: 2024-03-25T19:41:05.405 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 1 librados: starting msgr at 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 1 librados: starting objecter 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 1 Processor -- start 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 1 -- start start 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 1 librados: setting wanted keys 2024-03-25T19:41:05.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 1 librados: calling monclient init 2024-03-25T19:41:05.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.405+0000 7f96f4081640 10 monclient: init 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 10 monclient(hunting): picked mon.noname-a con 0x7f96ec0a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 10 monclient(hunting): picked mon.noname-c con 0x7f96ec13f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 10 monclient(hunting): picked mon.noname-b con 0x7f96ec1430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec1d7a90 con 0x7f96ec0a98f0 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0b1700 con 0x7f96ec1430c0 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0b13d0 con 0x7f96ec13f710 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:05.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.406+0000 7f96f4081640 10 monclient(hunting): authenticate will time out at 4236.547448s 2024-03-25T19:41:05.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.412+0000 7f96f261e640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f96ec1430c0 0x7f96ec14d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:41760/0 (socket says 172.21.15.110:41760) 2024-03-25T19:41:05.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.412+0000 7f96f261e640 1 -- 172.21.15.110:0/3251879229 learned_addr learned my addr 172.21.15.110:0/3251879229 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:05.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.416+0000 7f96e27fc640 10 client.?.objecter ms_handle_connect 0x7f96ec1430c0 2024-03-25T19:41:05.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.416+0000 7f96e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:05.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.423+0000 7f96e27fc640 10 client.?.objecter ms_handle_connect 0x7f96ec13f710 2024-03-25T19:41:05.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.423+0000 7f96e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:05.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.424+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076838402 0 0) 0x7f96ec0b1700 con 0x7f96ec1430c0 2024-03-25T19:41:05.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.424+0000 7f96e27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:05.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.424+0000 7f96e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:05.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.424+0000 7f96e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:05.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.424+0000 7f96e27fc640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:41:05.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.424+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d0004eb0 con 0x7f96ec1430c0 2024-03-25T19:41:05.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.427+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901681242 0 0) 0x7f96ec0b13d0 con 0x7f96ec13f710 2024-03-25T19:41:05.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.427+0000 7f96e27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:05.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.427+0000 7f96e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:05.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.427+0000 7f96e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:05.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.427+0000 7f96e27fc640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:41:05.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.427+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec0b1700 con 0x7f96ec13f710 2024-03-25T19:41:05.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.439+0000 7f96e27fc640 10 client.?.objecter ms_handle_connect 0x7f96ec0a98f0 2024-03-25T19:41:05.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.439+0000 7f96e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3181669799 0 0) 0x7f96d0004eb0 con 0x7f96ec1430c0 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ec0b13d0 con 0x7f96ec1430c0 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f96e4001c70 con 0x7f96ec1430c0 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 10 monclient(hunting): dump: 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:05.446 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869661082 0 0) 0x7f96ec0b1700 con 0x7f96ec13f710 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96d0004eb0 con 0x7f96ec13f710 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f96dc001b20 con 0x7f96ec13f710 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.446+0000 7f96e27fc640 10 monclient(hunting): dump: 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:05.448 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:05.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1259665564 0 0) 0x7f96d0004eb0 con 0x7f96ec13f710 2024-03-25T19:41:05.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 >> v1:172.21.15.178:6789/0 conn(0x7f96ec1430c0 legacy=0x7f96ec14d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 >> v1:172.21.15.110:6789/0 conn(0x7f96ec0a98f0 legacy=0x7f96ec1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:05.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 1 monclient: found mon.c 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec14eb50 con 0x7f96ec13f710 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 5 monclient: authenticate success, global_id 4259 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 10 monclient: _renew_subs 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec14dc50 con 0x7f96ec13f710 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 10 monclient: _renew_subs 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec14e290 con 0x7f96ec13f710 2024-03-25T19:41:05.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.454+0000 7f96f4081640 1 librados: init done 2024-03-25T19:41:05.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.455+0000 7f96f4081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:05.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.455+0000 7f96f4081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:05.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.455+0000 7f96f4081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:05.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.455+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec0b1560 con 0x7f96ec13f710 2024-03-25T19:41:05.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.678+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96dc0038a0 con 0x7f96ec13f710 2024-03-25T19:41:05.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.678+0000 7f96e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:05.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.678+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f96dc003a10 con 0x7f96ec13f710 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.678+0000 7f96e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.678+0000 7f96e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.678+0000 7f96e27fc640 10 monclient: dump: 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:05.678 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:05.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.678+0000 7f96e3fff640 4 set_mon_vals no callback set 2024-03-25T19:41:05.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.753+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f96dc0199e0 con 0x7f96ec13f710 2024-03-25T19:41:05.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.754+0000 7f96f1e1d640 10 monclient: get_auth_request con 0x7f96d006de80 auth_method 0 2024-03-25T19:41:05.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.754+0000 7f96e27fc640 10 client.4259.objecter ms_handle_connect 0x7f96d006de80 2024-03-25T19:41:05.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.761+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f96dc053520 con 0x7f96ec13f710 2024-03-25T19:41:05.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.761+0000 7f96e27fc640 10 client.4259.objecter ms_dispatch 0x7f96ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:05.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.761+0000 7f96e27fc640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:05.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.761+0000 7f96e27fc640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:05.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.761+0000 7f96e27fc640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-25T19:41:05.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.761+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96dc04f030 con 0x7f96ec13f710 2024-03-25T19:41:05.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.761+0000 7f96e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:05.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.761+0000 7f96e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:05.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.979+0000 7f96f4081640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:05.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.979+0000 7f96f4081640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:05.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.979+0000 7f96f4081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:05.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:05.979+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96ec0b0d00 con 0x7f96ec13f710 2024-03-25T19:41:06.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.005+0000 7f96e27fc640 1 -- 172.21.15.110:0/3251879229 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f96dc050f60 con 0x7f96ec13f710 2024-03-25T19:41:06.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.006+0000 7f96e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:06.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.006+0000 7f96e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:06.006 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:41:06.006 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 10 librados: watch_flush enter 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 10 librados: watch_flush exit 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 10 client.4259.objecter successfully canceled tick 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 >> v1:172.21.15.178:6800/1132739058 conn(0x7f96d006de80 legacy=0x7f96d0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 10 monclient: shutdown 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 >> v1:172.21.15.110:6790/0 conn(0x7f96ec13f710 legacy=0x7f96ec1d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 shutdown_connections 2024-03-25T19:41:06.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 >> 172.21.15.110:0/3251879229 conn(0x7f96ec0b7ea0 msgr2=0x7f96ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:06.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 shutdown_connections 2024-03-25T19:41:06.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 1 -- 172.21.15.110:0/3251879229 wait complete. 2024-03-25T19:41:06.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:06.009+0000 7f96f4081640 1 librados: shutdown 2024-03-25T19:41:06.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:12.828 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:12.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.944+0000 7fc788723640 10 monclient: get_monmap_and_config 2024-03-25T19:41:12.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.944+0000 7fc788723640 10 monclient: build_initial_monmap 2024-03-25T19:41:12.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.944+0000 7fc788723640 10 monclient: monmap: 2024-03-25T19:41:12.944 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:12.944 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:12.944 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:41:12.945312+0000 2024-03-25T19:41:12.944 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:41:12.945312+0000 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.944+0000 7fc788723640 1 Processor -- start 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 1 -- start start 2024-03-25T19:41:12.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 10 monclient: init 2024-03-25T19:41:12.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:12.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:12.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 10 monclient(hunting): picked mon.noname-b con 0x7fc7800a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:12.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 10 monclient(hunting): picked mon.noname-a con 0x7fc78013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:12.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 10 monclient(hunting): picked mon.noname-c con 0x7fc7801430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:12.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc780142520 con 0x7fc78013f710 2024-03-25T19:41:12.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7800b28a0 con 0x7fc7801430c0 2024-03-25T19:41:12.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7800b2a30 con 0x7fc7800a98f0 2024-03-25T19:41:12.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.945+0000 7fc788723640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:12.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.946+0000 7fc788723640 10 monclient(hunting): authenticate will time out at 4244.086832s 2024-03-25T19:41:12.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.963+0000 7fc786cc0640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fc7801430c0 0x7fc78014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:35116/0 (socket says 172.21.15.110:35116) 2024-03-25T19:41:12.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.963+0000 7fc786cc0640 1 -- 172.21.15.110:0/1091054648 learned_addr learned my addr 172.21.15.110:0/1091054648 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:12.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.980+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293839264 0 0) 0x7fc7800b28a0 con 0x7fc7801430c0 2024-03-25T19:41:12.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.980+0000 7fc7854bd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.980+0000 7fc7854bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.980+0000 7fc7854bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.980+0000 7fc7854bd640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:41:12.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.980+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76c003520 con 0x7fc7801430c0 2024-03-25T19:41:12.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.985+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163037465 0 0) 0x7fc780142520 con 0x7fc78013f710 2024-03-25T19:41:12.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.985+0000 7fc7854bd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.985+0000 7fc7854bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.985+0000 7fc7854bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.985+0000 7fc7854bd640 10 monclient(hunting): my global_id is 4251 2024-03-25T19:41:12.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.985+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7800b28a0 con 0x7fc78013f710 2024-03-25T19:41:12.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.986+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187720372 0 0) 0x7fc7800b2a30 con 0x7fc7800a98f0 2024-03-25T19:41:12.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.986+0000 7fc7854bd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.986+0000 7fc7854bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.986+0000 7fc7854bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.986+0000 7fc7854bd640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:41:12.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.986+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc780142520 con 0x7fc7800a98f0 2024-03-25T19:41:12.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520432419 0 0) 0x7fc76c003520 con 0x7fc7801430c0 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 >> v1:172.21.15.110:6790/0 conn(0x7fc7800a98f0 legacy=0x7fc7800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 >> v1:172.21.15.110:6789/0 conn(0x7fc78013f710 legacy=0x7fc7801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 1 monclient: found mon.noname-c 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7800b44c0 con 0x7fc7801430c0 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 10 monclient: _finish_auth 0 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc778001a10 con 0x7fc7801430c0 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc7854bd640 10 monclient: dump: 2024-03-25T19:41:12.991 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc788723640 5 monclient: authenticate success, global_id 4261 2024-03-25T19:41:12.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:12.991+0000 7fc788723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:13.016 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.017+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc778001b70 con 0x7fc7801430c0 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.017+0000 7fc7854bd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.017+0000 7fc7854bd640 1 -- 172.21.15.110:0/1091054648 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc778001f80 con 0x7fc7801430c0 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.017+0000 7fc7854bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.017+0000 7fc7854bd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.017+0000 7fc7854bd640 10 monclient: dump: 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:13.017 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:13.018 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:13.018 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:13.018 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:13.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.017+0000 7fc788723640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:13.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.017+0000 7fc788723640 4 set_mon_vals no callback set 2024-03-25T19:41:13.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.018+0000 7fc788723640 10 monclient: shutdown 2024-03-25T19:41:13.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.018+0000 7fc788723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:13.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.018+0000 7fc788723640 1 -- 172.21.15.110:0/1091054648 >> v1:172.21.15.178:6789/0 conn(0x7fc7801430c0 legacy=0x7fc78014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.018+0000 7fc788723640 1 -- 172.21.15.110:0/1091054648 shutdown_connections 2024-03-25T19:41:13.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.018+0000 7fc788723640 1 -- 172.21.15.110:0/1091054648 >> 172.21.15.110:0/1091054648 conn(0x7fc7800b7ea0 msgr2=0x7fc7800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:13.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.018+0000 7fc788723640 1 -- 172.21.15.110:0/1091054648 shutdown_connections 2024-03-25T19:41:13.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.018+0000 7fc788723640 1 -- 172.21.15.110:0/1091054648 wait complete. 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 10 monclient: build_initial_monmap 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 10 monclient: monmap: 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 1 librados: starting msgr at 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 1 librados: starting objecter 2024-03-25T19:41:13.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 1 Processor -- start 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 1 -- start start 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 1 librados: setting wanted keys 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 1 librados: calling monclient init 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.019+0000 7fc788723640 10 monclient: init 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 10 monclient(hunting): picked mon.noname-b con 0x7fc7800a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 10 monclient(hunting): picked mon.noname-c con 0x7fc78013f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 10 monclient(hunting): picked mon.noname-a con 0x7fc7801430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc780136a10 con 0x7fc7801430c0 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc780133b60 con 0x7fc7800a98f0 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7800ab950 con 0x7fc78013f710 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:13.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.020+0000 7fc788723640 10 monclient(hunting): authenticate will time out at 4244.161207s 2024-03-25T19:41:13.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.031+0000 7fc7864bf640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fc7800a98f0 0x7fc780132d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:35166/0 (socket says 172.21.15.110:35166) 2024-03-25T19:41:13.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.031+0000 7fc7864bf640 1 -- 172.21.15.110:0/3221974379 learned_addr learned my addr 172.21.15.110:0/3221974379 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:13.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.032+0000 7fc776ffd640 10 client.?.objecter ms_handle_connect 0x7fc7800a98f0 2024-03-25T19:41:13.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.032+0000 7fc776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:13.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.034+0000 7fc776ffd640 10 client.?.objecter ms_handle_connect 0x7fc78013f710 2024-03-25T19:41:13.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.034+0000 7fc776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:13.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.039+0000 7fc776ffd640 10 client.?.objecter ms_handle_connect 0x7fc7801430c0 2024-03-25T19:41:13.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.039+0000 7fc776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:13.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.045+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976938472 0 0) 0x7fc7800ab950 con 0x7fc78013f710 2024-03-25T19:41:13.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.045+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:13.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.045+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:13.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.045+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:13.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.045+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:41:13.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.045+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc764004eb0 con 0x7fc78013f710 2024-03-25T19:41:13.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.054+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666321119 0 0) 0x7fc780133b60 con 0x7fc7800a98f0 2024-03-25T19:41:13.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.054+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:13.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.054+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:13.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.054+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:13.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.054+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:41:13.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.054+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7800ab950 con 0x7fc7800a98f0 2024-03-25T19:41:13.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.057+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327499962 0 0) 0x7fc780136a10 con 0x7fc7801430c0 2024-03-25T19:41:13.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.057+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:13.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.057+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:13.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.057+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:13.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.057+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:41:13.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.057+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc780133b60 con 0x7fc7801430c0 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.065+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4080821741 0 0) 0x7fc764004eb0 con 0x7fc78013f710 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.065+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc780136a10 con 0x7fc78013f710 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.065+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc770001970 con 0x7fc78013f710 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.065+0000 7fc776ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.065+0000 7fc776ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.065+0000 7fc776ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:13.065 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:13.066 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:13.066 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:13.066 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:13.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.065+0000 7fc776ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:13.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.068+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3284542642 0 0) 0x7fc7800ab950 con 0x7fc7800a98f0 2024-03-25T19:41:13.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.068+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc764004eb0 con 0x7fc7800a98f0 2024-03-25T19:41:13.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.068+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc77c0019f0 con 0x7fc7800a98f0 2024-03-25T19:41:13.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.068+0000 7fc776ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:13.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.068+0000 7fc776ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:13.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.068+0000 7fc776ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:13.069 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:13.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.071+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 248996945 0 0) 0x7fc780133b60 con 0x7fc7801430c0 2024-03-25T19:41:13.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.071+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7800ab950 con 0x7fc7801430c0 2024-03-25T19:41:13.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.071+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc778001e80 con 0x7fc7801430c0 2024-03-25T19:41:13.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.071+0000 7fc776ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:13.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.071+0000 7fc776ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:13.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.071+0000 7fc776ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:13.071 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:13.071 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:13.072 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.072 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.072 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:13.072 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:13.072 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:13.072 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:13.072 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:13.072 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:13.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2307778715 0 0) 0x7fc780136a10 con 0x7fc78013f710 2024-03-25T19:41:13.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 >> v1:172.21.15.178:6789/0 conn(0x7fc7800a98f0 legacy=0x7fc780132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 >> v1:172.21.15.110:6789/0 conn(0x7fc7801430c0 legacy=0x7fc780133450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 1 monclient: found mon.c 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7800abc30 con 0x7fc78013f710 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 5 monclient: authenticate success, global_id 4265 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 10 monclient: _renew_subs 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7800abf10 con 0x7fc78013f710 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:13.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 10 monclient: _renew_subs 2024-03-25T19:41:13.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:13.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc780137e00 con 0x7fc78013f710 2024-03-25T19:41:13.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.080+0000 7fc788723640 1 librados: init done 2024-03-25T19:41:13.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.081+0000 7fc788723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:13.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.081+0000 7fc788723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:13.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.081+0000 7fc788723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:13.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.081+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7800627b0 con 0x7fc78013f710 2024-03-25T19:41:13.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.379+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7700035f0 con 0x7fc78013f710 2024-03-25T19:41:13.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.379+0000 7fc776ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.379+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fc770003900 con 0x7fc78013f710 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.379+0000 7fc776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.379+0000 7fc776ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.379+0000 7fc776ffd640 10 monclient: dump: 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:13.383 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:13.384 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:13.384 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:13.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.379+0000 7fc784cbc640 4 set_mon_vals no callback set 2024-03-25T19:41:13.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.380+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fc770019950 con 0x7fc78013f710 2024-03-25T19:41:13.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.381+0000 7fc7864bf640 10 monclient: get_auth_request con 0x7fc76406e0f0 auth_method 0 2024-03-25T19:41:13.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.381+0000 7fc776ffd640 10 client.4265.objecter ms_handle_connect 0x7fc76406e0f0 2024-03-25T19:41:13.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.385+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fc770053600 con 0x7fc78013f710 2024-03-25T19:41:13.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.385+0000 7fc776ffd640 10 client.4265.objecter ms_dispatch 0x7fc780000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:13.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.385+0000 7fc776ffd640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:13.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.385+0000 7fc776ffd640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:13.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.385+0000 7fc776ffd640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-25T19:41:13.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.385+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc77004f110 con 0x7fc78013f710 2024-03-25T19:41:13.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.385+0000 7fc776ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:13.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.385+0000 7fc776ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:13.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.635+0000 7fc788723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.635+0000 7fc788723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.635+0000 7fc788723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:13.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.635+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc78013c060 con 0x7fc78013f710 2024-03-25T19:41:13.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.663+0000 7fc776ffd640 1 -- 172.21.15.110:0/3221974379 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fc770051040 con 0x7fc78013f710 2024-03-25T19:41:13.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.663+0000 7fc776ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.663+0000 7fc776ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:13.663 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:41:13.663 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:13.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.665+0000 7fc788723640 10 librados: watch_flush enter 2024-03-25T19:41:13.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 10 librados: watch_flush exit 2024-03-25T19:41:13.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-25T19:41:13.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 10 client.4265.objecter successfully canceled tick 2024-03-25T19:41:13.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 >> v1:172.21.15.178:6800/1132739058 conn(0x7fc76406e0f0 legacy=0x7fc7640705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 10 monclient: shutdown 2024-03-25T19:41:13.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:13.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 >> v1:172.21.15.110:6790/0 conn(0x7fc78013f710 legacy=0x7fc780136300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 shutdown_connections 2024-03-25T19:41:13.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 >> 172.21.15.110:0/3221974379 conn(0x7fc7800b7ea0 msgr2=0x7fc780074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:13.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 shutdown_connections 2024-03-25T19:41:13.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 1 -- 172.21.15.110:0/3221974379 wait complete. 2024-03-25T19:41:13.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:13.666+0000 7fc788723640 1 librados: shutdown 2024-03-25T19:41:13.678 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:16.815 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:24.100 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:16.875 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:24.160 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:16.910 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:24.165 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:16.966 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:24.221 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:17.020 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:24.304 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:17.083 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:24.367 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:17.112 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:24.396 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:17.197 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:24.451 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:17.323 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:24.608 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:20.485 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:20.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.601+0000 7f147547b640 10 monclient: get_monmap_and_config 2024-03-25T19:41:20.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.601+0000 7f147547b640 10 monclient: build_initial_monmap 2024-03-25T19:41:20.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.601+0000 7f147547b640 10 monclient: monmap: 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:41:20.601999+0000 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:41:20.601999+0000 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:41:20.601 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:20.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.601+0000 7f147547b640 1 Processor -- start 2024-03-25T19:41:20.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 1 -- start start 2024-03-25T19:41:20.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 10 monclient: init 2024-03-25T19:41:20.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:20.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:20.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 10 monclient(hunting): picked mon.noname-c con 0x7f14700a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:20.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 10 monclient(hunting): picked mon.noname-a con 0x7f147013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:20.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 10 monclient(hunting): picked mon.noname-b con 0x7f14701430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:20.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1470142520 con 0x7f147013f710 2024-03-25T19:41:20.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700b28a0 con 0x7f14700a98f0 2024-03-25T19:41:20.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700b2a30 con 0x7f14701430c0 2024-03-25T19:41:20.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:20.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.602+0000 7f147547b640 10 monclient(hunting): authenticate will time out at 4251.743681s 2024-03-25T19:41:20.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.616+0000 7f146effd640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f14700a98f0 0x7f14700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:39340/0 (socket says 172.21.15.110:39340) 2024-03-25T19:41:20.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.616+0000 7f146effd640 1 -- 172.21.15.110:0/172343275 learned_addr learned my addr 172.21.15.110:0/172343275 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:20.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.633+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082938635 0 0) 0x7f14700b28a0 con 0x7f14700a98f0 2024-03-25T19:41:20.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.633+0000 7f146e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.633+0000 7f146e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.633+0000 7f146e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.633+0000 7f146e7fc640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:41:20.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.633+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f145c003520 con 0x7f14700a98f0 2024-03-25T19:41:20.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.642+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737877925 0 0) 0x7f1470142520 con 0x7f147013f710 2024-03-25T19:41:20.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.642+0000 7f146e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.642+0000 7f146e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.642+0000 7f146e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.642+0000 7f146e7fc640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:41:20.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.642+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14700b28a0 con 0x7f147013f710 2024-03-25T19:41:20.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1303200139 0 0) 0x7f145c003520 con 0x7f14700a98f0 2024-03-25T19:41:20.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 >> v1:172.21.15.110:6790/0 conn(0x7f14701430c0 legacy=0x7f147014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 >> v1:172.21.15.110:6789/0 conn(0x7f147013f710 legacy=0x7f14701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14700b44c0 con 0x7f14700a98f0 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1460001690 con 0x7f14700a98f0 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f146e7fc640 10 monclient: dump: 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:20.644 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:20.645 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.645 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:20.645 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:20.645 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:20.645 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:20.645 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:20.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f147547b640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:41:20.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.644+0000 7f147547b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:20.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.652+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1460001870 con 0x7f14700a98f0 2024-03-25T19:41:20.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.652+0000 7f146e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.652+0000 7f146e7fc640 1 -- 172.21.15.110:0/172343275 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1460001c80 con 0x7f14700a98f0 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.652+0000 7f146e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.652+0000 7f146e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.652+0000 7f146e7fc640 10 monclient: dump: 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.652+0000 7f147547b640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:20.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.652+0000 7f147547b640 4 set_mon_vals no callback set 2024-03-25T19:41:20.653 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.653+0000 7f147547b640 10 monclient: shutdown 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.653+0000 7f147547b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.653+0000 7f147547b640 1 -- 172.21.15.110:0/172343275 >> v1:172.21.15.178:6789/0 conn(0x7f14700a98f0 legacy=0x7f14700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.653+0000 7f147547b640 1 -- 172.21.15.110:0/172343275 shutdown_connections 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.653+0000 7f147547b640 1 -- 172.21.15.110:0/172343275 >> 172.21.15.110:0/172343275 conn(0x7f14700b7ea0 msgr2=0x7f14700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.653+0000 7f147547b640 1 -- 172.21.15.110:0/172343275 shutdown_connections 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.653+0000 7f147547b640 1 -- 172.21.15.110:0/172343275 wait complete. 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.654+0000 7f147547b640 10 monclient: build_initial_monmap 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.654+0000 7f147547b640 10 monclient: monmap: 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:20.654 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.654+0000 7f147547b640 1 librados: starting msgr at 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.654+0000 7f147547b640 1 librados: starting objecter 2024-03-25T19:41:20.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.654+0000 7f147547b640 1 Processor -- start 2024-03-25T19:41:20.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.655+0000 7f147547b640 1 -- start start 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.655+0000 7f147547b640 1 librados: setting wanted keys 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.655+0000 7f147547b640 1 librados: calling monclient init 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.655+0000 7f147547b640 10 monclient: init 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.655+0000 7f147547b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.655+0000 7f147547b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.655+0000 7f147547b640 10 monclient(hunting): picked mon.noname-b con 0x7f14700a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.656+0000 7f147547b640 10 monclient(hunting): picked mon.noname-a con 0x7f147013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.656+0000 7f147547b640 10 monclient(hunting): picked mon.noname-c con 0x7f14701430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.656+0000 7f147547b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700abc00 con 0x7f147013f710 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.656+0000 7f147547b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1470137ac0 con 0x7f14700a98f0 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.656+0000 7f147547b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14701d77f0 con 0x7f14701430c0 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.656+0000 7f147547b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.656+0000 7f147547b640 10 monclient(hunting): authenticate will time out at 4251.796973s 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.657+0000 7f14667fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f147013f710 0x7f14701d7d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50830/0 (socket says 172.21.15.110:50830) 2024-03-25T19:41:20.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.657+0000 7f14667fc640 1 -- 172.21.15.110:0/25024698 learned_addr learned my addr 172.21.15.110:0/25024698 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:20.658 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.659+0000 7f1467fff640 10 client.?.objecter ms_handle_connect 0x7f147013f710 2024-03-25T19:41:20.658 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.659+0000 7f1467fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.668+0000 7f1467fff640 10 client.?.objecter ms_handle_connect 0x7f14700a98f0 2024-03-25T19:41:20.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.668+0000 7f1467fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.671+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756257812 0 0) 0x7f14700abc00 con 0x7f147013f710 2024-03-25T19:41:20.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.671+0000 7f1467fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.671+0000 7f1467fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.671+0000 7f1467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.671+0000 7f1467fff640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:41:20.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.671+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1450004eb0 con 0x7f147013f710 2024-03-25T19:41:20.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.677+0000 7f1467fff640 10 client.?.objecter ms_handle_connect 0x7f14701430c0 2024-03-25T19:41:20.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.677+0000 7f1467fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.689 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.690+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037322227 0 0) 0x7f1470137ac0 con 0x7f14700a98f0 2024-03-25T19:41:20.689 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.690+0000 7f1467fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.689 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.690+0000 7f1467fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.690+0000 7f1467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.690+0000 7f1467fff640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:41:20.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.690+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14700abc00 con 0x7f14700a98f0 2024-03-25T19:41:20.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.698+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738271019 0 0) 0x7f14701d77f0 con 0x7f14701430c0 2024-03-25T19:41:20.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.698+0000 7f1467fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.698+0000 7f1467fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.698+0000 7f1467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.698+0000 7f1467fff640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:41:20.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.698+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470137ac0 con 0x7f14701430c0 2024-03-25T19:41:20.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.703+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3104458129 0 0) 0x7f14700abc00 con 0x7f14700a98f0 2024-03-25T19:41:20.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.703+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14701d77f0 con 0x7f14700a98f0 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.703+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1460001b80 con 0x7f14700a98f0 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.703+0000 7f1467fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.703+0000 7f1467fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.703+0000 7f1467fff640 10 monclient(hunting): dump: 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:20.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.703+0000 7f1467fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:20.705 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.706+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895108912 0 0) 0x7f1450004eb0 con 0x7f147013f710 2024-03-25T19:41:20.705 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.706+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14700abc00 con 0x7f147013f710 2024-03-25T19:41:20.705 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.706+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1458001a10 con 0x7f147013f710 2024-03-25T19:41:20.705 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.706+0000 7f1467fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.706+0000 7f1467fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.706+0000 7f1467fff640 10 monclient(hunting): dump: 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:20.706 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:20.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 395475880 0 0) 0x7f14700abc00 con 0x7f147013f710 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 >> v1:172.21.15.110:6790/0 conn(0x7f14701430c0 legacy=0x7f14701d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 >> v1:172.21.15.178:6789/0 conn(0x7f14700a98f0 legacy=0x7f1470138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 1 monclient: found mon.a 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470138c20 con 0x7f147013f710 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f1467fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 5 monclient: authenticate success, global_id 4260 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 10 monclient: _renew_subs 2024-03-25T19:41:20.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:20.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1470138f00 con 0x7f147013f710 2024-03-25T19:41:20.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:20.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 10 monclient: _renew_subs 2024-03-25T19:41:20.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:20.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701d43c0 con 0x7f147013f710 2024-03-25T19:41:20.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.714+0000 7f147547b640 1 librados: init done 2024-03-25T19:41:20.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.715+0000 7f147547b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:20.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.715+0000 7f147547b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:20.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.715+0000 7f147547b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:20.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:20.715+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14700627b0 con 0x7f147013f710 2024-03-25T19:41:21.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.039+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1458001c50 con 0x7f147013f710 2024-03-25T19:41:21.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.039+0000 7f1467fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:21.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.039+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f14580038c0 con 0x7f147013f710 2024-03-25T19:41:21.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.039+0000 7f1467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:21.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.039+0000 7f1467fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:21.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.039+0000 7f1467fff640 10 monclient: dump: 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:21.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.039+0000 7f146dffb640 4 set_mon_vals no callback set 2024-03-25T19:41:21.041 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.040+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f1458003aa0 con 0x7f147013f710 2024-03-25T19:41:21.041 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.041+0000 7f146effd640 10 monclient: get_auth_request con 0x7f145006dfb0 auth_method 0 2024-03-25T19:41:21.041 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.041+0000 7f1467fff640 10 client.4260.objecter ms_handle_connect 0x7f145006dfb0 2024-03-25T19:41:21.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.046+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f1458054750 con 0x7f147013f710 2024-03-25T19:41:21.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.046+0000 7f1467fff640 10 client.4260.objecter ms_dispatch 0x7f1470000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:21.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.046+0000 7f1467fff640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:21.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.046+0000 7f1467fff640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:21.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.046+0000 7f1467fff640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-25T19:41:21.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.046+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1458050260 con 0x7f147013f710 2024-03-25T19:41:21.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.046+0000 7f1467fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:21.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.046+0000 7f1467fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:21.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.294+0000 7f147547b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:21.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.294+0000 7f147547b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:21.294 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.294+0000 7f147547b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:21.294 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.294+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f147013e230 con 0x7f147013f710 2024-03-25T19:41:21.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.336+0000 7f1467fff640 1 -- 172.21.15.110:0/25024698 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f1458052190 con 0x7f147013f710 2024-03-25T19:41:21.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.336+0000 7f1467fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:21.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.336+0000 7f1467fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:21.336 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:41:21.336 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:21.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 10 librados: watch_flush enter 2024-03-25T19:41:21.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 10 librados: watch_flush exit 2024-03-25T19:41:21.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-25T19:41:21.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 10 client.4260.objecter successfully canceled tick 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 >> v1:172.21.15.178:6800/1132739058 conn(0x7f145006dfb0 legacy=0x7f1450070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 10 monclient: shutdown 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 >> v1:172.21.15.110:6789/0 conn(0x7f147013f710 legacy=0x7f14701d7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 shutdown_connections 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 >> 172.21.15.110:0/25024698 conn(0x7f14700b7ea0 msgr2=0x7f1470074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 shutdown_connections 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.338+0000 7f147547b640 1 -- 172.21.15.110:0/25024698 wait complete. 2024-03-25T19:41:21.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:21.339+0000 7f147547b640 1 librados: shutdown 2024-03-25T19:41:21.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:28.157 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:28.274 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.274+0000 7f81e7c55640 10 monclient: get_monmap_and_config 2024-03-25T19:41:28.274 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.274+0000 7f81e7c55640 10 monclient: build_initial_monmap 2024-03-25T19:41:28.274 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.274+0000 7f81e7c55640 10 monclient: monmap: 2024-03-25T19:41:28.274 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:41:28.275887+0000 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:41:28.275887+0000 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:41:28.275 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:28.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.275+0000 7f81e7c55640 1 Processor -- start 2024-03-25T19:41:28.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.275+0000 7f81e7c55640 1 -- start start 2024-03-25T19:41:28.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.275+0000 7f81e7c55640 10 monclient: init 2024-03-25T19:41:28.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:28.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:28.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 10 monclient(hunting): picked mon.noname-b con 0x7f81e00a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:28.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 10 monclient(hunting): picked mon.noname-a con 0x7f81e013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:28.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 10 monclient(hunting): picked mon.noname-c con 0x7f81e01430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:28.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0142520 con 0x7f81e013f710 2024-03-25T19:41:28.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00b28a0 con 0x7f81e01430c0 2024-03-25T19:41:28.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00b2a30 con 0x7f81e00a98f0 2024-03-25T19:41:28.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:28.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.276+0000 7f81e7c55640 10 monclient(hunting): authenticate will time out at 4259.417419s 2024-03-25T19:41:28.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.284+0000 7f81e61f2640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f81e01430c0 0x7f81e014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:51136/0 (socket says 172.21.15.110:51136) 2024-03-25T19:41:28.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.284+0000 7f81e61f2640 1 -- 172.21.15.110:0/477159631 learned_addr learned my addr 172.21.15.110:0/477159631 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:28.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.293+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104262350 0 0) 0x7f81e00b28a0 con 0x7f81e01430c0 2024-03-25T19:41:28.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.293+0000 7f81e49ef640 10 monclient(hunting): _negotiate 2024-03-25T19:41:28.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.293+0000 7f81e49ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:28.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.293+0000 7f81e49ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:28.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.293+0000 7f81e49ef640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:41:28.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.293+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc003520 con 0x7f81e01430c0 2024-03-25T19:41:28.297 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.298+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628001620 0 0) 0x7f81e0142520 con 0x7f81e013f710 2024-03-25T19:41:28.298 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.298+0000 7f81e49ef640 10 monclient(hunting): _negotiate 2024-03-25T19:41:28.298 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.298+0000 7f81e49ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:28.298 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.298+0000 7f81e49ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:28.298 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.298+0000 7f81e49ef640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:41:28.298 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.298+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e00b28a0 con 0x7f81e013f710 2024-03-25T19:41:28.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.309+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750139728 0 0) 0x7f81e00b2a30 con 0x7f81e00a98f0 2024-03-25T19:41:28.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.309+0000 7f81e49ef640 10 monclient(hunting): _negotiate 2024-03-25T19:41:28.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.309+0000 7f81e49ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:28.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.309+0000 7f81e49ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:28.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.309+0000 7f81e49ef640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:41:28.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.309+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e0142520 con 0x7f81e00a98f0 2024-03-25T19:41:28.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 567171094 0 0) 0x7f81cc003520 con 0x7f81e01430c0 2024-03-25T19:41:28.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 >> v1:172.21.15.110:6790/0 conn(0x7f81e00a98f0 legacy=0x7f81e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 >> v1:172.21.15.110:6789/0 conn(0x7f81e013f710 legacy=0x7f81e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 10 monclient: _finish_hunting 0 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 1 monclient: found mon.noname-c 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e00b44c0 con 0x7f81e01430c0 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 10 monclient: _finish_auth 0 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 10 monclient: _check_auth_tickets 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:28.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81d8001a10 con 0x7f81e01430c0 2024-03-25T19:41:28.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:28.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:28.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e49ef640 10 monclient: dump: 2024-03-25T19:41:28.320 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:28.320 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:28.320 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.320 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.320 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:28.321 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:28.321 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:28.321 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:28.321 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:28.321 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:28.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e7c55640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:41:28.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.318+0000 7f81e7c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:28.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.334+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d8001b70 con 0x7f81e01430c0 2024-03-25T19:41:28.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.334+0000 7f81e49ef640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:28.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.334+0000 7f81e49ef640 1 -- 172.21.15.110:0/477159631 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81d8001f80 con 0x7f81e01430c0 2024-03-25T19:41:28.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.334+0000 7f81e49ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:28.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.334+0000 7f81e49ef640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:28.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.334+0000 7f81e49ef640 10 monclient: dump: 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.334+0000 7f81e7c55640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:28.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.334+0000 7f81e7c55640 4 set_mon_vals no callback set 2024-03-25T19:41:28.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.335+0000 7f81e7c55640 10 monclient: shutdown 2024-03-25T19:41:28.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.335+0000 7f81e7c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:28.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.335+0000 7f81e7c55640 1 -- 172.21.15.110:0/477159631 >> v1:172.21.15.178:6789/0 conn(0x7f81e01430c0 legacy=0x7f81e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.335+0000 7f81e7c55640 1 -- 172.21.15.110:0/477159631 shutdown_connections 2024-03-25T19:41:28.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.335+0000 7f81e7c55640 1 -- 172.21.15.110:0/477159631 >> 172.21.15.110:0/477159631 conn(0x7f81e00b7ea0 msgr2=0x7f81e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:28.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.335+0000 7f81e7c55640 1 -- 172.21.15.110:0/477159631 shutdown_connections 2024-03-25T19:41:28.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.336+0000 7f81e7c55640 1 -- 172.21.15.110:0/477159631 wait complete. 2024-03-25T19:41:28.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.336+0000 7f81e7c55640 10 monclient: build_initial_monmap 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.336+0000 7f81e7c55640 10 monclient: monmap: 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 librados: starting msgr at 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 librados: starting objecter 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 Processor -- start 2024-03-25T19:41:28.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 -- start start 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 librados: setting wanted keys 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 librados: calling monclient init 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 10 monclient: init 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 10 monclient(hunting): picked mon.noname-a con 0x7f81e00a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 10 monclient(hunting): picked mon.noname-b con 0x7f81e013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 10 monclient(hunting): picked mon.noname-c con 0x7f81e01430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0136ec0 con 0x7f81e00a98f0 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0136c20 con 0x7f81e013f710 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0137db0 con 0x7f81e01430c0 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:28.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.337+0000 7f81e7c55640 10 monclient(hunting): authenticate will time out at 4259.478681s 2024-03-25T19:41:28.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.344+0000 7f81e59f1640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f81e00a98f0 0x7f81e01376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58500/0 (socket says 172.21.15.110:58500) 2024-03-25T19:41:28.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.344+0000 7f81e59f1640 1 -- 172.21.15.110:0/1927042263 learned_addr learned my addr 172.21.15.110:0/1927042263 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:28.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.346+0000 7f81d67fc640 10 client.?.objecter ms_handle_connect 0x7f81e00a98f0 2024-03-25T19:41:28.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.346+0000 7f81d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:28.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.359+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550875307 0 0) 0x7f81e0136ec0 con 0x7f81e00a98f0 2024-03-25T19:41:28.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.359+0000 7f81d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:28.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.359+0000 7f81d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:28.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.359+0000 7f81d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:28.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.359+0000 7f81d67fc640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:41:28.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.359+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c4004eb0 con 0x7f81e00a98f0 2024-03-25T19:41:28.360 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.360+0000 7f81d67fc640 10 client.?.objecter ms_handle_connect 0x7f81e01430c0 2024-03-25T19:41:28.360 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.360+0000 7f81d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:28.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.367+0000 7f81d67fc640 10 client.?.objecter ms_handle_connect 0x7f81e013f710 2024-03-25T19:41:28.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.367+0000 7f81d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:28.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.373+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578574144 0 0) 0x7f81e0136c20 con 0x7f81e013f710 2024-03-25T19:41:28.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.373+0000 7f81d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:28.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.373+0000 7f81d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:28.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.373+0000 7f81d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:28.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.373+0000 7f81d67fc640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:41:28.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.373+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e0136ec0 con 0x7f81e013f710 2024-03-25T19:41:28.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.376+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413175613 0 0) 0x7f81e0137db0 con 0x7f81e01430c0 2024-03-25T19:41:28.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.376+0000 7f81d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:28.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.376+0000 7f81d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:28.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.376+0000 7f81d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:28.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.376+0000 7f81d67fc640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:41:28.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.376+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e0136c20 con 0x7f81e01430c0 2024-03-25T19:41:28.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.380+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343023116 0 0) 0x7f81c4004eb0 con 0x7f81e00a98f0 2024-03-25T19:41:28.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.380+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e0137db0 con 0x7f81e00a98f0 2024-03-25T19:41:28.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.380+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81dc001a10 con 0x7f81e00a98f0 2024-03-25T19:41:28.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.380+0000 7f81d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:28.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.380+0000 7f81d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:28.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.380+0000 7f81d67fc640 10 monclient(hunting): dump: 2024-03-25T19:41:28.381 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:28.381 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:28.382 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.382 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.382 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:28.382 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:28.382 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:28.382 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:28.383 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:28.383 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:28.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.380+0000 7f81d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.389+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 203500826 0 0) 0x7f81e0136c20 con 0x7f81e01430c0 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.389+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4004eb0 con 0x7f81e01430c0 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.389+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81d8001e80 con 0x7f81e01430c0 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.389+0000 7f81d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.389+0000 7f81d67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.389+0000 7f81d67fc640 10 monclient(hunting): dump: 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.389 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.390 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:28.390 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:28.390 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:28.390 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:28.390 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:28.390 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:28.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.392+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 564009536 0 0) 0x7f81e0136ec0 con 0x7f81e013f710 2024-03-25T19:41:28.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.393+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e0136c20 con 0x7f81e013f710 2024-03-25T19:41:28.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.393+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81d0001970 con 0x7f81e013f710 2024-03-25T19:41:28.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.393+0000 7f81d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:28.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.393+0000 7f81d67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:28.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.393+0000 7f81d67fc640 10 monclient(hunting): dump: 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:28.394 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:28.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 969463052 0 0) 0x7f81c4004eb0 con 0x7f81e01430c0 2024-03-25T19:41:28.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 >> v1:172.21.15.178:6789/0 conn(0x7f81e013f710 legacy=0x7f81e00b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 >> v1:172.21.15.110:6789/0 conn(0x7f81e00a98f0 legacy=0x7f81e01376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:28.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 1 monclient: found mon.c 2024-03-25T19:41:28.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:28.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:28.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e0138b50 con 0x7f81e01430c0 2024-03-25T19:41:28.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:28.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:28.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:28.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 5 monclient: authenticate success, global_id 4277 2024-03-25T19:41:28.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 10 monclient: _renew_subs 2024-03-25T19:41:28.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:28.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e0139830 con 0x7f81e01430c0 2024-03-25T19:41:28.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:28.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 10 monclient: _renew_subs 2024-03-25T19:41:28.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:28.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e01369b0 con 0x7f81e01430c0 2024-03-25T19:41:28.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.399+0000 7f81e7c55640 1 librados: init done 2024-03-25T19:41:28.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.400+0000 7f81e7c55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:28.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.400+0000 7f81e7c55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:28.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.400+0000 7f81e7c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:28.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.400+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e014f320 con 0x7f81e01430c0 2024-03-25T19:41:28.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.698+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d8001560 con 0x7f81e01430c0 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.698+0000 7f81d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.698+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81d8004e10 con 0x7f81e01430c0 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.698+0000 7f81d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.698+0000 7f81d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.698+0000 7f81d67fc640 10 monclient: dump: 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:28.698 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:28.699 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:28.699 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:28.699 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:28.699 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:28.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.698+0000 7f81d7fff640 4 set_mon_vals no callback set 2024-03-25T19:41:28.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.699+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f81d801ade0 con 0x7f81e01430c0 2024-03-25T19:41:28.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.699+0000 7f81e59f1640 10 monclient: get_auth_request con 0x7f81c406e160 auth_method 0 2024-03-25T19:41:28.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.699+0000 7f81d67fc640 10 client.4277.objecter ms_handle_connect 0x7f81c406e160 2024-03-25T19:41:28.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.703+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f81d8054990 con 0x7f81e01430c0 2024-03-25T19:41:28.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.703+0000 7f81d67fc640 10 client.4277.objecter ms_dispatch 0x7f81e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:28.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.703+0000 7f81d67fc640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:28.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.703+0000 7f81d67fc640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:28.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.703+0000 7f81d67fc640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T19:41:28.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.703+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81d80502d0 con 0x7f81e01430c0 2024-03-25T19:41:28.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.703+0000 7f81d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:28.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.703+0000 7f81d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:28.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.952+0000 7f81e7c55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.952+0000 7f81e7c55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.952+0000 7f81e7c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:41:28.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.952+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81e00627b0 con 0x7f81e01430c0 2024-03-25T19:41:28.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.991+0000 7f81d67fc640 1 -- 172.21.15.110:0/1927042263 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f81d80523d0 con 0x7f81e01430c0 2024-03-25T19:41:28.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.991+0000 7f81d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.991+0000 7f81d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:28.991 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:41:28.991 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 10 librados: watch_flush enter 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 10 librados: watch_flush exit 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 10 client.4277.objecter successfully canceled tick 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 >> v1:172.21.15.178:6800/1132739058 conn(0x7f81c406e160 legacy=0x7f81c4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 10 monclient: shutdown 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 >> v1:172.21.15.110:6790/0 conn(0x7f81e01430c0 legacy=0x7f81e01e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 shutdown_connections 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.993+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 >> 172.21.15.110:0/1927042263 conn(0x7f81e00b7ea0 msgr2=0x7f81e013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.994+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 shutdown_connections 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.994+0000 7f81e7c55640 1 -- 172.21.15.110:0/1927042263 wait complete. 2024-03-25T19:41:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:28.994+0000 7f81e7c55640 1 librados: shutdown 2024-03-25T19:41:29.004 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:35.810 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.925+0000 7f9afefe6640 10 monclient: get_monmap_and_config 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.925+0000 7f9afefe6640 10 monclient: build_initial_monmap 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.925+0000 7f9afefe6640 10 monclient: monmap: 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:41:35.926660+0000 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:41:35.926660+0000 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:35.925 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:35.926 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:35.926 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:41:35.926 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:41:35.926 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.926+0000 7f9afefe6640 1 Processor -- start 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.926+0000 7f9afefe6640 1 -- start start 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.926+0000 7f9afefe6640 10 monclient: init 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 10 monclient(hunting): picked mon.noname-b con 0x7f9af80a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 10 monclient(hunting): picked mon.noname-c con 0x7f9af813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 10 monclient(hunting): picked mon.noname-a con 0x7f9af81430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:35.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af8142520 con 0x7f9af81430c0 2024-03-25T19:41:35.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af80b28a0 con 0x7f9af813f710 2024-03-25T19:41:35.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af80b2a30 con 0x7f9af80a98f0 2024-03-25T19:41:35.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:35.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.927+0000 7f9afefe6640 10 monclient(hunting): authenticate will time out at 4267.068606s 2024-03-25T19:41:35.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.937+0000 7f9af7fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f9af813f710 0x7f9af81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:42788/0 (socket says 172.21.15.110:42788) 2024-03-25T19:41:35.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.937+0000 7f9af7fff640 1 -- 172.21.15.110:0/1092326536 learned_addr learned my addr 172.21.15.110:0/1092326536 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:35.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.956+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328212250 0 0) 0x7f9af80b28a0 con 0x7f9af813f710 2024-03-25T19:41:35.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.956+0000 7f9af77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.956+0000 7f9af77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.956+0000 7f9af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.956+0000 7f9af77fe640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:41:35.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.956+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae8003520 con 0x7f9af813f710 2024-03-25T19:41:35.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.962+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812616814 0 0) 0x7f9af80b2a30 con 0x7f9af80a98f0 2024-03-25T19:41:35.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.962+0000 7f9af77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.962+0000 7f9af77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.962+0000 7f9af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.962+0000 7f9af77fe640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:41:35.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.962+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af80b28a0 con 0x7f9af80a98f0 2024-03-25T19:41:35.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593832403 0 0) 0x7f9ae8003520 con 0x7f9af813f710 2024-03-25T19:41:35.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 >> v1:172.21.15.110:6790/0 conn(0x7f9af80a98f0 legacy=0x7f9af80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 >> v1:172.21.15.110:6789/0 conn(0x7f9af81430c0 legacy=0x7f9af814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 1 monclient: found mon.noname-c 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af80b44c0 con 0x7f9af813f710 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9ae4001690 con 0x7f9af813f710 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9afefe6640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9afefe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:35.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9af77fe640 10 monclient: dump: 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:35.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.964+0000 7f9afefe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:35.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.995+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ae40017f0 con 0x7f9af813f710 2024-03-25T19:41:35.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.995+0000 7f9af77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:35.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.995+0000 7f9af77fe640 1 -- 172.21.15.110:0/1092326536 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9ae4001c00 con 0x7f9af813f710 2024-03-25T19:41:35.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.995+0000 7f9afefe6640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:35.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.995+0000 7f9afefe6640 4 set_mon_vals no callback set 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.995+0000 7f9af77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.995+0000 7f9af77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.995+0000 7f9af77fe640 10 monclient: dump: 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:35.996 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:35.997 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.996+0000 7f9afefe6640 10 monclient: shutdown 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.996+0000 7f9afefe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.996+0000 7f9afefe6640 1 -- 172.21.15.110:0/1092326536 >> v1:172.21.15.178:6789/0 conn(0x7f9af813f710 legacy=0x7f9af81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.996+0000 7f9afefe6640 1 -- 172.21.15.110:0/1092326536 shutdown_connections 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.996+0000 7f9afefe6640 1 -- 172.21.15.110:0/1092326536 >> 172.21.15.110:0/1092326536 conn(0x7f9af80b7ea0 msgr2=0x7f9af80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.996+0000 7f9afefe6640 1 -- 172.21.15.110:0/1092326536 shutdown_connections 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.996+0000 7f9afefe6640 1 -- 172.21.15.110:0/1092326536 wait complete. 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.997+0000 7f9afefe6640 10 monclient: build_initial_monmap 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.997+0000 7f9afefe6640 10 monclient: monmap: 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:35.999 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.997+0000 7f9afefe6640 1 librados: starting msgr at 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.997+0000 7f9afefe6640 1 librados: starting objecter 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.998+0000 7f9afefe6640 1 Processor -- start 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.998+0000 7f9afefe6640 1 -- start start 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.998+0000 7f9afefe6640 1 librados: setting wanted keys 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.998+0000 7f9afefe6640 1 librados: calling monclient init 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.998+0000 7f9afefe6640 10 monclient: init 2024-03-25T19:41:36.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 10 monclient(hunting): picked mon.noname-b con 0x7f9af80a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 10 monclient(hunting): picked mon.noname-a con 0x7f9af813f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 10 monclient(hunting): picked mon.noname-c con 0x7f9af81430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af8138eb0 con 0x7f9af813f710 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af8139120 con 0x7f9af80a98f0 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af8132c50 con 0x7f9af81430c0 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:36.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:35.999+0000 7f9afefe6640 10 monclient(hunting): authenticate will time out at 4267.139947s 2024-03-25T19:41:36.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.005+0000 7f9afcd82640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f9af80a98f0 0x7f9af8137b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:42830/0 (socket says 172.21.15.110:42830) 2024-03-25T19:41:36.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.005+0000 7f9afcd82640 1 -- 172.21.15.110:0/1380472560 learned_addr learned my addr 172.21.15.110:0/1380472560 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:36.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.006+0000 7f9af57fa640 10 client.?.objecter ms_handle_connect 0x7f9af80a98f0 2024-03-25T19:41:36.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.006+0000 7f9af57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:36.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.009+0000 7f9af57fa640 10 client.?.objecter ms_handle_connect 0x7f9af81430c0 2024-03-25T19:41:36.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.009+0000 7f9af57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:36.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.015+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386292164 0 0) 0x7f9af8139120 con 0x7f9af80a98f0 2024-03-25T19:41:36.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.015+0000 7f9af57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:36.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.015+0000 7f9af57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:36.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.015+0000 7f9af57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:36.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.015+0000 7f9af57fa640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:41:36.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.015+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc004eb0 con 0x7f9af80a98f0 2024-03-25T19:41:36.023 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.023+0000 7f9af57fa640 10 client.?.objecter ms_handle_connect 0x7f9af813f710 2024-03-25T19:41:36.023 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.023+0000 7f9af57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:36.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.027+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960849037 0 0) 0x7f9af8132c50 con 0x7f9af81430c0 2024-03-25T19:41:36.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.027+0000 7f9af57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:36.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.027+0000 7f9af57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:36.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.027+0000 7f9af57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:36.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.027+0000 7f9af57fa640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:41:36.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.027+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af8139120 con 0x7f9af81430c0 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.031+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3441077959 0 0) 0x7f9adc004eb0 con 0x7f9af80a98f0 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.031+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9af8132c50 con 0x7f9af80a98f0 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.031+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9aec001a10 con 0x7f9af80a98f0 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.031+0000 7f9af57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.031+0000 7f9af57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.031+0000 7f9af57fa640 10 monclient(hunting): dump: 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:36.031 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:36.032 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:36.032 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:36.032 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.031+0000 7f9af57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:36.032 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.032+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958430963 0 0) 0x7f9af8138eb0 con 0x7f9af813f710 2024-03-25T19:41:36.032 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.032+0000 7f9af57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:36.032 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.032+0000 7f9af57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:36.032 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.032+0000 7f9af57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:36.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.032+0000 7f9af57fa640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:41:36.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.032+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc004eb0 con 0x7f9af813f710 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1094651664 0 0) 0x7f9af8132c50 con 0x7f9af80a98f0 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 >> v1:172.21.15.110:6790/0 conn(0x7f9af81430c0 legacy=0x7f9af8132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 >> v1:172.21.15.110:6789/0 conn(0x7f9af813f710 legacy=0x7f9af8131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 1 monclient: found mon.b 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af8139400 con 0x7f9af80a98f0 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:36.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9af57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 10 monclient: _renew_subs 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af8137640 con 0x7f9af80a98f0 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 10 monclient: _renew_subs 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af81382a0 con 0x7f9af80a98f0 2024-03-25T19:41:36.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.043+0000 7f9afefe6640 1 librados: init done 2024-03-25T19:41:36.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.044+0000 7f9afefe6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:36.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.044+0000 7f9afefe6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:36.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.044+0000 7f9afefe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:36.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.044+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af80627b0 con 0x7f9af80a98f0 2024-03-25T19:41:36.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.345+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aec001c50 con 0x7f9af80a98f0 2024-03-25T19:41:36.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.345+0000 7f9af57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:36.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.345+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9aec003a60 con 0x7f9af80a98f0 2024-03-25T19:41:36.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.345+0000 7f9af57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:36.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.345+0000 7f9af57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:36.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.345+0000 7f9af57fa640 10 monclient: dump: 2024-03-25T19:41:36.345 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:36.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.345+0000 7f9af6ffd640 4 set_mon_vals no callback set 2024-03-25T19:41:36.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.347+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f9aec003c40 con 0x7f9af80a98f0 2024-03-25T19:41:36.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.347+0000 7f9af7fff640 10 monclient: get_auth_request con 0x7f9adc06de00 auth_method 0 2024-03-25T19:41:36.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.347+0000 7f9af57fa640 10 client.4282.objecter ms_handle_connect 0x7f9adc06de00 2024-03-25T19:41:36.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.350+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f9aec054900 con 0x7f9af80a98f0 2024-03-25T19:41:36.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.350+0000 7f9af57fa640 10 client.4282.objecter ms_dispatch 0x7f9af8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:36.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.350+0000 7f9af57fa640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:36.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.350+0000 7f9af57fa640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:36.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.350+0000 7f9af57fa640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T19:41:36.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.351+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9aec050410 con 0x7f9af80a98f0 2024-03-25T19:41:36.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.351+0000 7f9af57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:36.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.351+0000 7f9af57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:36.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.604+0000 7f9afefe6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.604+0000 7f9afefe6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.604+0000 7f9afefe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:36.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.604+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9af813e250 con 0x7f9af80a98f0 2024-03-25T19:41:36.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.636+0000 7f9af57fa640 1 -- 172.21.15.110:0/1380472560 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f9aec052340 con 0x7f9af80a98f0 2024-03-25T19:41:36.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.636+0000 7f9af57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.636+0000 7f9af57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:36.636 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:41:36.636 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.638+0000 7f9afefe6640 10 librados: watch_flush enter 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.638+0000 7f9afefe6640 10 librados: watch_flush exit 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.638+0000 7f9afefe6640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.638+0000 7f9afefe6640 10 client.4282.objecter successfully canceled tick 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 >> v1:172.21.15.178:6800/1132739058 conn(0x7f9adc06de00 legacy=0x7f9adc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 10 monclient: shutdown 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 >> v1:172.21.15.178:6789/0 conn(0x7f9af80a98f0 legacy=0x7f9af8137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 shutdown_connections 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 >> 172.21.15.110:0/1380472560 conn(0x7f9af80b7ea0 msgr2=0x7f9af8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 shutdown_connections 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 1 -- 172.21.15.110:0/1380472560 wait complete. 2024-03-25T19:41:36.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:36.639+0000 7f9afefe6640 1 librados: shutdown 2024-03-25T19:41:36.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:36.810 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:44.094 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.866 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:44.151 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.911 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:44.195 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.921 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:44.176 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.922 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:44.206 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.926 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:44.180 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.927 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:44.212 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.975 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:01:44.230 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.978 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:44.263 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:43.457 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:43.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.571+0000 7f1a69b02640 10 monclient: get_monmap_and_config 2024-03-25T19:41:43.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.571+0000 7f1a69b02640 10 monclient: build_initial_monmap 2024-03-25T19:41:43.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.571+0000 7f1a69b02640 10 monclient: monmap: 2024-03-25T19:41:43.571 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:41:43.573126+0000 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:41:43.573126+0000 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:41:43.572 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:43.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.572+0000 7f1a69b02640 1 Processor -- start 2024-03-25T19:41:43.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.572+0000 7f1a69b02640 1 -- start start 2024-03-25T19:41:43.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.572+0000 7f1a69b02640 10 monclient: init 2024-03-25T19:41:43.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.572+0000 7f1a69b02640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:43.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.572+0000 7f1a69b02640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:43.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.573+0000 7f1a69b02640 10 monclient(hunting): picked mon.noname-a con 0x7f1a640a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:43.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.573+0000 7f1a69b02640 10 monclient(hunting): picked mon.noname-c con 0x7f1a6413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:43.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.573+0000 7f1a69b02640 10 monclient(hunting): picked mon.noname-b con 0x7f1a641430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:43.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.573+0000 7f1a69b02640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a64142520 con 0x7f1a640a98f0 2024-03-25T19:41:43.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.573+0000 7f1a69b02640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a640b28a0 con 0x7f1a6413f710 2024-03-25T19:41:43.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.573+0000 7f1a69b02640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a640b2a30 con 0x7f1a641430c0 2024-03-25T19:41:43.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.573+0000 7f1a69b02640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.573+0000 7f1a69b02640 10 monclient(hunting): authenticate will time out at 4274.715151s 2024-03-25T19:41:43.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.582+0000 7f1a63fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f1a641430c0 0x7f1a6414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36690/0 (socket says 172.21.15.110:36690) 2024-03-25T19:41:43.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.582+0000 7f1a63fff640 1 -- 172.21.15.110:0/561205630 learned_addr learned my addr 172.21.15.110:0/561205630 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:43.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.600+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478700084 0 0) 0x7f1a640b2a30 con 0x7f1a641430c0 2024-03-25T19:41:43.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.600+0000 7f1a627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.600+0000 7f1a627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.600+0000 7f1a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.600+0000 7f1a627fc640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:41:43.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.600+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a4c003520 con 0x7f1a641430c0 2024-03-25T19:41:43.610 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311023111 0 0) 0x7f1a640b28a0 con 0x7f1a6413f710 2024-03-25T19:41:43.610 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.610 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.610 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.610 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:41:43.610 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a640b2a30 con 0x7f1a6413f710 2024-03-25T19:41:43.611 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85992822 0 0) 0x7f1a64142520 con 0x7f1a640a98f0 2024-03-25T19:41:43.611 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.611 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.611 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.611 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:41:43.611 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.610+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a640b28a0 con 0x7f1a640a98f0 2024-03-25T19:41:43.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1697759334 0 0) 0x7f1a640b2a30 con 0x7f1a6413f710 2024-03-25T19:41:43.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 >> v1:172.21.15.110:6790/0 conn(0x7f1a641430c0 legacy=0x7f1a6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 >> v1:172.21.15.110:6789/0 conn(0x7f1a640a98f0 legacy=0x7f1a640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:43.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 1 monclient: found mon.noname-c 2024-03-25T19:41:43.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:43.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:41:43.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a640b44c0 con 0x7f1a6413f710 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1a58001670 con 0x7f1a6413f710 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a69b02640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a69b02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a627fc640 10 monclient: dump: 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.622 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:43.623 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:43.623 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:43.623 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:43.623 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:43.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.620+0000 7f1a69b02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:43.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.630+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a580017d0 con 0x7f1a6413f710 2024-03-25T19:41:43.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.630+0000 7f1a627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:43.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.630+0000 7f1a627fc640 1 -- 172.21.15.110:0/561205630 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1a58001be0 con 0x7f1a6413f710 2024-03-25T19:41:43.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.630+0000 7f1a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.630+0000 7f1a627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:43.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.630+0000 7f1a627fc640 10 monclient: dump: 2024-03-25T19:41:43.631 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:43.631 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.630+0000 7f1a69b02640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:43.632 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.630+0000 7f1a69b02640 4 set_mon_vals no callback set 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.631+0000 7f1a69b02640 10 monclient: shutdown 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.631+0000 7f1a69b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.631+0000 7f1a69b02640 1 -- 172.21.15.110:0/561205630 >> v1:172.21.15.178:6789/0 conn(0x7f1a6413f710 legacy=0x7f1a641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.631+0000 7f1a69b02640 1 -- 172.21.15.110:0/561205630 shutdown_connections 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.631+0000 7f1a69b02640 1 -- 172.21.15.110:0/561205630 >> 172.21.15.110:0/561205630 conn(0x7f1a640b7ea0 msgr2=0x7f1a640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.632+0000 7f1a69b02640 1 -- 172.21.15.110:0/561205630 shutdown_connections 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.632+0000 7f1a69b02640 1 -- 172.21.15.110:0/561205630 wait complete. 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.632+0000 7f1a69b02640 10 monclient: build_initial_monmap 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.632+0000 7f1a69b02640 10 monclient: monmap: 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:43.634 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.633+0000 7f1a69b02640 1 librados: starting msgr at 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.633+0000 7f1a69b02640 1 librados: starting objecter 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.633+0000 7f1a69b02640 1 Processor -- start 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.633+0000 7f1a69b02640 1 -- start start 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.633+0000 7f1a69b02640 1 librados: setting wanted keys 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.633+0000 7f1a69b02640 1 librados: calling monclient init 2024-03-25T19:41:43.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.633+0000 7f1a69b02640 10 monclient: init 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 10 monclient(hunting): picked mon.noname-c con 0x7f1a640a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 10 monclient(hunting): picked mon.noname-b con 0x7f1a6413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 10 monclient(hunting): picked mon.noname-a con 0x7f1a641430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a641daaf0 con 0x7f1a641430c0 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a64136e50 con 0x7f1a6413f710 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a64137130 con 0x7f1a640a98f0 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.634+0000 7f1a69b02640 10 monclient(hunting): authenticate will time out at 4274.776204s 2024-03-25T19:41:43.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.642+0000 7f1a637fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f1a640a98f0 0x7f1a641daf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36752/0 (socket says 172.21.15.110:36752) 2024-03-25T19:41:43.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.642+0000 7f1a637fe640 1 -- 172.21.15.110:0/2781725981 learned_addr learned my addr 172.21.15.110:0/2781725981 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:43.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.642+0000 7f1a4bfff640 10 client.?.objecter ms_handle_connect 0x7f1a640a98f0 2024-03-25T19:41:43.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.642+0000 7f1a4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.655+0000 7f1a4bfff640 10 client.?.objecter ms_handle_connect 0x7f1a6413f710 2024-03-25T19:41:43.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.655+0000 7f1a4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.669+0000 7f1a4bfff640 10 client.?.objecter ms_handle_connect 0x7f1a641430c0 2024-03-25T19:41:43.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.669+0000 7f1a4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.678 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.678+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328461920 0 0) 0x7f1a641daaf0 con 0x7f1a641430c0 2024-03-25T19:41:43.678 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.678+0000 7f1a4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.678+0000 7f1a4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.678+0000 7f1a4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.678+0000 7f1a4bfff640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:41:43.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.678+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a50004eb0 con 0x7f1a641430c0 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.747+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028618954 0 0) 0x7f1a50004eb0 con 0x7f1a641430c0 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.747+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a641daaf0 con 0x7f1a641430c0 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.747+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1a5c001af0 con 0x7f1a641430c0 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.747+0000 7f1a4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.747+0000 7f1a4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.747+0000 7f1a4bfff640 10 monclient(hunting): dump: 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:43.748 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.749 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:43.749 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:43.749 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:43.749 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:43.749 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:43.749 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.747+0000 7f1a4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2590770711 0 0) 0x7f1a641daaf0 con 0x7f1a641430c0 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 >> v1:172.21.15.110:6790/0 conn(0x7f1a640a98f0 legacy=0x7f1a641daf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 >> v1:172.21.15.178:6789/0 conn(0x7f1a6413f710 legacy=0x7f1a64135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 1 monclient: found mon.a 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a64139f80 con 0x7f1a641430c0 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 5 monclient: authenticate success, global_id 4278 2024-03-25T19:41:43.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 10 monclient: _renew_subs 2024-03-25T19:41:43.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:43.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a641db690 con 0x7f1a641430c0 2024-03-25T19:41:43.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:43.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 10 monclient: _renew_subs 2024-03-25T19:41:43.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:43.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a64135c60 con 0x7f1a641430c0 2024-03-25T19:41:43.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.760+0000 7f1a69b02640 1 librados: init done 2024-03-25T19:41:43.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.761+0000 7f1a69b02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.761+0000 7f1a69b02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.761+0000 7f1a69b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:43.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:43.761+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a640b3ef0 con 0x7f1a641430c0 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.076+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a5c001f60 con 0x7f1a641430c0 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.076+0000 7f1a4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.076+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1a5c004d00 con 0x7f1a641430c0 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.076+0000 7f1a4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.076+0000 7f1a4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.076+0000 7f1a4bfff640 10 monclient: dump: 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:44.077 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:44.078 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.078 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:44.078 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:44.078 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:44.078 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:44.078 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:44.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.076+0000 7f1a61ffb640 4 set_mon_vals no callback set 2024-03-25T19:41:44.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.077+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f1a5c004f20 con 0x7f1a641430c0 2024-03-25T19:41:44.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.078+0000 7f1a637fe640 10 monclient: get_auth_request con 0x7f1a5006dc70 auth_method 0 2024-03-25T19:41:44.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.078+0000 7f1a4bfff640 10 client.4278.objecter ms_handle_connect 0x7f1a5006dc70 2024-03-25T19:41:44.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.083+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f1a5c0549b0 con 0x7f1a641430c0 2024-03-25T19:41:44.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.083+0000 7f1a4bfff640 10 client.4278.objecter ms_dispatch 0x7f1a64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:44.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.083+0000 7f1a4bfff640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:44.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.083+0000 7f1a4bfff640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:44.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.083+0000 7f1a4bfff640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-25T19:41:44.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.083+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a5c0504c0 con 0x7f1a641430c0 2024-03-25T19:41:44.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.083+0000 7f1a4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:44.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.083+0000 7f1a4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:44.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.338+0000 7f1a69b02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:44.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.338+0000 7f1a69b02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:44.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.338+0000 7f1a69b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:44.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.338+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a64136c90 con 0x7f1a641430c0 2024-03-25T19:41:44.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.398+0000 7f1a4bfff640 1 -- 172.21.15.110:0/2781725981 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f1a5c0523f0 con 0x7f1a641430c0 2024-03-25T19:41:44.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.398+0000 7f1a4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:44.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.398+0000 7f1a4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:44.399 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:41:44.399 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:44.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 10 librados: watch_flush enter 2024-03-25T19:41:44.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 10 librados: watch_flush exit 2024-03-25T19:41:44.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-25T19:41:44.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 10 client.4278.objecter successfully canceled tick 2024-03-25T19:41:44.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 >> v1:172.21.15.178:6800/1132739058 conn(0x7f1a5006dc70 legacy=0x7f1a50070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 10 monclient: shutdown 2024-03-25T19:41:44.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:44.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 >> v1:172.21.15.110:6789/0 conn(0x7f1a641430c0 legacy=0x7f1a641e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 shutdown_connections 2024-03-25T19:41:44.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.400+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 >> 172.21.15.110:0/2781725981 conn(0x7f1a640b7ea0 msgr2=0x7f1a64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:44.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.401+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 shutdown_connections 2024-03-25T19:41:44.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.401+0000 7f1a69b02640 1 -- 172.21.15.110:0/2781725981 wait complete. 2024-03-25T19:41:44.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:44.401+0000 7f1a69b02640 1 librados: shutdown 2024-03-25T19:41:44.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:51.218 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:51.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.336+0000 7f1dd3fc8640 10 monclient: get_monmap_and_config 2024-03-25T19:41:51.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.336+0000 7f1dd3fc8640 10 monclient: build_initial_monmap 2024-03-25T19:41:51.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.336+0000 7f1dd3fc8640 10 monclient: monmap: 2024-03-25T19:41:51.336 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:51.336 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:51.336 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:41:51.337685+0000 2024-03-25T19:41:51.336 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:41:51.337685+0000 2024-03-25T19:41:51.336 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:51.337 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:51.337 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:51.337 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:41:51.337 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:41:51.337 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.337+0000 7f1dd3fc8640 1 Processor -- start 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.337+0000 7f1dd3fc8640 1 -- start start 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.337+0000 7f1dd3fc8640 10 monclient: init 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f1dcc0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f1dcc13f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f1dcc1430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc142520 con 0x7f1dcc13f710 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc0b28a0 con 0x7f1dcc0a98f0 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc0b2a30 con 0x7f1dcc1430c0 2024-03-25T19:41:51.338 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:51.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.338+0000 7f1dd3fc8640 10 monclient(hunting): authenticate will time out at 4282.479570s 2024-03-25T19:41:51.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.348+0000 7f1dd1563640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1dcc13f710 0x7f1dcc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44024/0 (socket says 172.21.15.110:44024) 2024-03-25T19:41:51.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.348+0000 7f1dd1563640 1 -- 172.21.15.110:0/1799496154 learned_addr learned my addr 172.21.15.110:0/1799496154 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135651662 0 0) 0x7f1dcc142520 con 0x7f1dcc13f710 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 10 monclient(hunting): _negotiate 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db8003520 con 0x7f1dcc13f710 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104924147 0 0) 0x7f1dcc0b2a30 con 0x7f1dcc1430c0 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 10 monclient(hunting): _negotiate 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:41:51.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.363+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc142520 con 0x7f1dcc1430c0 2024-03-25T19:41:51.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.373+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587990800 0 0) 0x7f1dcc0b28a0 con 0x7f1dcc0a98f0 2024-03-25T19:41:51.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.373+0000 7f1dd0d62640 10 monclient(hunting): _negotiate 2024-03-25T19:41:51.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.373+0000 7f1dd0d62640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:51.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.373+0000 7f1dd0d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:51.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.373+0000 7f1dd0d62640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:41:51.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.373+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc0b2a30 con 0x7f1dcc0a98f0 2024-03-25T19:41:51.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835632136 0 0) 0x7f1dcc142520 con 0x7f1dcc1430c0 2024-03-25T19:41:51.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 >> v1:172.21.15.178:6789/0 conn(0x7f1dcc0a98f0 legacy=0x7f1dcc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 >> v1:172.21.15.110:6789/0 conn(0x7f1dcc13f710 legacy=0x7f1dcc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 10 monclient: _finish_hunting 0 2024-03-25T19:41:51.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 1 monclient: found mon.noname-b 2024-03-25T19:41:51.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:51.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc0b44c0 con 0x7f1dcc1430c0 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 10 monclient: _finish_auth 0 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 10 monclient: _check_auth_tickets 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1dc40019f0 con 0x7f1dcc1430c0 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd0d62640 10 monclient: dump: 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:51.382 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:51.383 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:51.383 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:51.383 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:51.383 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:51.383 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:51.383 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:51.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd3fc8640 5 monclient: authenticate success, global_id 4292 2024-03-25T19:41:51.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.381+0000 7f1dd3fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:51.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.399+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dc4001b50 con 0x7f1dcc1430c0 2024-03-25T19:41:51.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.399+0000 7f1dd0d62640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:51.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.399+0000 7f1dd0d62640 1 -- 172.21.15.110:0/1799496154 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1dc4001f60 con 0x7f1dcc1430c0 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.399+0000 7f1dd0d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.399+0000 7f1dd0d62640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.399+0000 7f1dd0d62640 10 monclient: dump: 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:51.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.399+0000 7f1dd3fc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:51.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.399+0000 7f1dd3fc8640 4 set_mon_vals no callback set 2024-03-25T19:41:51.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.400+0000 7f1dd3fc8640 10 monclient: shutdown 2024-03-25T19:41:51.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.400+0000 7f1dd3fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:51.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.400+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/1799496154 >> v1:172.21.15.110:6790/0 conn(0x7f1dcc1430c0 legacy=0x7f1dcc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.400+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/1799496154 shutdown_connections 2024-03-25T19:41:51.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.400+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/1799496154 >> 172.21.15.110:0/1799496154 conn(0x7f1dcc0b7ea0 msgr2=0x7f1dcc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:51.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.400+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/1799496154 shutdown_connections 2024-03-25T19:41:51.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.400+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/1799496154 wait complete. 2024-03-25T19:41:51.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.401+0000 7f1dd3fc8640 10 monclient: build_initial_monmap 2024-03-25T19:41:51.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.401+0000 7f1dd3fc8640 10 monclient: monmap: 2024-03-25T19:41:51.402 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:51.402 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.402+0000 7f1dd3fc8640 1 librados: starting msgr at 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.402+0000 7f1dd3fc8640 1 librados: starting objecter 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.402+0000 7f1dd3fc8640 1 Processor -- start 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.402+0000 7f1dd3fc8640 1 -- start start 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.402+0000 7f1dd3fc8640 1 librados: setting wanted keys 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.402+0000 7f1dd3fc8640 1 librados: calling monclient init 2024-03-25T19:41:51.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.402+0000 7f1dd3fc8640 10 monclient: init 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f1dcc0a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f1dcc13f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f1dcc1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc0abac0 con 0x7f1dcc1430c0 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc1d4310 con 0x7f1dcc13f710 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc138f10 con 0x7f1dcc0a98f0 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:51.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.403+0000 7f1dd3fc8640 10 monclient(hunting): authenticate will time out at 4282.544048s 2024-03-25T19:41:51.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.417+0000 7f1dd1563640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f1dcc13f710 0x7f1dcc134080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:53786/0 (socket says 172.21.15.110:53786) 2024-03-25T19:41:51.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.417+0000 7f1dd1563640 1 -- 172.21.15.110:0/3550689455 learned_addr learned my addr 172.21.15.110:0/3550689455 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:51.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.419+0000 7f1dc27fc640 10 client.?.objecter ms_handle_connect 0x7f1dcc13f710 2024-03-25T19:41:51.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.419+0000 7f1dc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:51.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.421+0000 7f1dc27fc640 10 client.?.objecter ms_handle_connect 0x7f1dcc0a98f0 2024-03-25T19:41:51.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.421+0000 7f1dc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:51.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.429+0000 7f1dc27fc640 10 client.?.objecter ms_handle_connect 0x7f1dcc1430c0 2024-03-25T19:41:51.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.429+0000 7f1dc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:51.430 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.431+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067248700 0 0) 0x7f1dcc138f10 con 0x7f1dcc0a98f0 2024-03-25T19:41:51.430 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.431+0000 7f1dc27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:51.430 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.431+0000 7f1dc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:51.430 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.431+0000 7f1dc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:51.430 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.431+0000 7f1dc27fc640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:41:51.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.431+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db0004eb0 con 0x7f1dcc0a98f0 2024-03-25T19:41:51.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.435+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774858273 0 0) 0x7f1dcc1d4310 con 0x7f1dcc13f710 2024-03-25T19:41:51.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.435+0000 7f1dc27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:51.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.435+0000 7f1dc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:51.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.435+0000 7f1dc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:51.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.435+0000 7f1dc27fc640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:41:51.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.435+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc138f10 con 0x7f1dcc13f710 2024-03-25T19:41:51.443 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.444+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352614817 0 0) 0x7f1dcc0abac0 con 0x7f1dcc1430c0 2024-03-25T19:41:51.444 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.444+0000 7f1dc27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:51.444 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.444+0000 7f1dc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:51.444 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.444+0000 7f1dc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:51.444 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.444+0000 7f1dc27fc640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:41:51.444 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.444+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc1d4310 con 0x7f1dcc1430c0 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.445+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2597290274 0 0) 0x7f1dcc138f10 con 0x7f1dcc13f710 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.445+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dcc0abac0 con 0x7f1dcc13f710 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.445+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1dbc0018d0 con 0x7f1dcc13f710 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.445+0000 7f1dc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.445+0000 7f1dc27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.445+0000 7f1dc27fc640 10 monclient(hunting): dump: 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:51.445 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:51.446 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:51.446 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:51.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.445+0000 7f1dc27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2160657114 0 0) 0x7f1dcc0abac0 con 0x7f1dcc13f710 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 >> v1:172.21.15.110:6790/0 conn(0x7f1dcc0a98f0 legacy=0x7f1dcc138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 >> v1:172.21.15.110:6789/0 conn(0x7f1dcc1430c0 legacy=0x7f1dcc1d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 1 monclient: found mon.b 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc135010 con 0x7f1dcc13f710 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:51.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 10 monclient: _renew_subs 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc134790 con 0x7f1dcc13f710 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 10 monclient: _renew_subs 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc1d5120 con 0x7f1dcc13f710 2024-03-25T19:41:51.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.462+0000 7f1dd3fc8640 1 librados: init done 2024-03-25T19:41:51.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.463+0000 7f1dd3fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:51.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.463+0000 7f1dd3fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:51.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.463+0000 7f1dd3fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:51.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.463+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc1d44a0 con 0x7f1dcc13f710 2024-03-25T19:41:51.743 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.744+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dbc001d00 con 0x7f1dcc13f710 2024-03-25T19:41:51.743 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.744+0000 7f1dc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:51.743 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.744+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1dbc004980 con 0x7f1dcc13f710 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.744+0000 7f1dc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.744+0000 7f1dc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.744+0000 7f1dc27fc640 10 monclient: dump: 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:51.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.744+0000 7f1dc3fff640 4 set_mon_vals no callback set 2024-03-25T19:41:51.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.745+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f1dbc004ba0 con 0x7f1dcc13f710 2024-03-25T19:41:51.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.746+0000 7f1dd1d64640 10 monclient: get_auth_request con 0x7f1db006ddd0 auth_method 0 2024-03-25T19:41:51.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.746+0000 7f1dc27fc640 10 client.4294.objecter ms_handle_connect 0x7f1db006ddd0 2024-03-25T19:41:51.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.747+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f1dbc0547e0 con 0x7f1dcc13f710 2024-03-25T19:41:51.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.747+0000 7f1dc27fc640 10 client.4294.objecter ms_dispatch 0x7f1dcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:51.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.747+0000 7f1dc27fc640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:51.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.747+0000 7f1dc27fc640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:51.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.747+0000 7f1dc27fc640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T19:41:51.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.776+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dbc050120 con 0x7f1dcc13f710 2024-03-25T19:41:51.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.776+0000 7f1dc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:51.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:51.776+0000 7f1dc27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:52.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.031+0000 7f1dd3fc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:52.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.031+0000 7f1dd3fc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:52.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.031+0000 7f1dd3fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:41:52.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.031+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dcc1351a0 con 0x7f1dcc13f710 2024-03-25T19:41:52.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.079+0000 7f1dc27fc640 1 -- 172.21.15.110:0/3550689455 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f1dbc052220 con 0x7f1dcc13f710 2024-03-25T19:41:52.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.079+0000 7f1dc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:52.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.079+0000 7f1dc27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:52.079 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:41:52.079 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 10 librados: watch_flush enter 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 10 librados: watch_flush exit 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 10 client.4294.objecter successfully canceled tick 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 >> v1:172.21.15.178:6800/1132739058 conn(0x7f1db006ddd0 legacy=0x7f1db0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 10 monclient: shutdown 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 >> v1:172.21.15.178:6789/0 conn(0x7f1dcc13f710 legacy=0x7f1dcc134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 shutdown_connections 2024-03-25T19:41:52.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.081+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 >> 172.21.15.110:0/3550689455 conn(0x7f1dcc0b7ea0 msgr2=0x7f1dcc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:52.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.082+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 shutdown_connections 2024-03-25T19:41:52.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.082+0000 7f1dd3fc8640 1 -- 172.21.15.110:0/3550689455 wait complete. 2024-03-25T19:41:52.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:52.082+0000 7f1dd3fc8640 1 librados: shutdown 2024-03-25T19:41:52.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:56.993 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:04.277 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:57.003 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:04.288 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:57.046 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:04.330 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:57.050 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:04.305 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:57.061 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:04.316 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:57.065 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:04.349 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:57.078 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:04.363 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:57.084 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:04.339 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:57.108 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:04.392 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:58.901 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:59.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.015+0000 7f858e85c640 10 monclient: get_monmap_and_config 2024-03-25T19:41:59.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.015+0000 7f858e85c640 10 monclient: build_initial_monmap 2024-03-25T19:41:59.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.015+0000 7f858e85c640 10 monclient: monmap: 2024-03-25T19:41:59.015 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:41:59.016835+0000 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:41:59.016835+0000 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:41:59.016 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:59.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.016+0000 7f858e85c640 1 Processor -- start 2024-03-25T19:41:59.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 1 -- start start 2024-03-25T19:41:59.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 10 monclient: init 2024-03-25T19:41:59.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:59.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:59.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 10 monclient(hunting): picked mon.noname-c con 0x7f85880a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:59.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 10 monclient(hunting): picked mon.noname-a con 0x7f858813f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:59.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 10 monclient(hunting): picked mon.noname-b con 0x7f85881430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:59.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8588142520 con 0x7f858813f710 2024-03-25T19:41:59.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85880b28a0 con 0x7f85880a98f0 2024-03-25T19:41:59.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85880b2a30 con 0x7f85881430c0 2024-03-25T19:41:59.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:59.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.017+0000 7f858e85c640 10 monclient(hunting): authenticate will time out at 4290.158671s 2024-03-25T19:41:59.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.024+0000 7f8587fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f85880a98f0 0x7f85880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:38904/0 (socket says 172.21.15.110:38904) 2024-03-25T19:41:59.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.025+0000 7f8587fff640 1 -- 172.21.15.110:0/3389900528 learned_addr learned my addr 172.21.15.110:0/3389900528 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:59.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.035+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020004713 0 0) 0x7f85880b2a30 con 0x7f85881430c0 2024-03-25T19:41:59.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.035+0000 7f8586ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:59.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.035+0000 7f8586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:59.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.035+0000 7f8586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:59.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.035+0000 7f8586ffd640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:41:59.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.035+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8570003520 con 0x7f85881430c0 2024-03-25T19:41:59.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.037+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423658230 0 0) 0x7f8588142520 con 0x7f858813f710 2024-03-25T19:41:59.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.037+0000 7f8586ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:59.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.037+0000 7f8586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:59.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.037+0000 7f8586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:59.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.037+0000 7f8586ffd640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:41:59.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.037+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85880b2a30 con 0x7f858813f710 2024-03-25T19:41:59.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069789049 0 0) 0x7f8570003520 con 0x7f85881430c0 2024-03-25T19:41:59.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 >> v1:172.21.15.178:6789/0 conn(0x7f85880a98f0 legacy=0x7f85880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 >> v1:172.21.15.110:6789/0 conn(0x7f858813f710 legacy=0x7f85881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 1 monclient: found mon.noname-b 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85880b44c0 con 0x7f85881430c0 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8580001a10 con 0x7f85881430c0 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f858e85c640 5 monclient: authenticate success, global_id 4298 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f858e85c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:59.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f8586ffd640 10 monclient: dump: 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:59.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.048+0000 7f858e85c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:59.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.057+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8580001b70 con 0x7f85881430c0 2024-03-25T19:41:59.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.057+0000 7f8586ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:59.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.057+0000 7f8586ffd640 1 -- 172.21.15.110:0/3389900528 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8580001f80 con 0x7f85881430c0 2024-03-25T19:41:59.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.057+0000 7f858e85c640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:59.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.057+0000 7f858e85c640 4 set_mon_vals no callback set 2024-03-25T19:41:59.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.057+0000 7f8586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:59.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.057+0000 7f8586ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:59.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.057+0000 7f8586ffd640 10 monclient: dump: 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:59.058 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:59.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.058+0000 7f858e85c640 10 monclient: shutdown 2024-03-25T19:41:59.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.058+0000 7f858e85c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:59.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.058+0000 7f858e85c640 1 -- 172.21.15.110:0/3389900528 >> v1:172.21.15.110:6790/0 conn(0x7f85881430c0 legacy=0x7f858814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.058+0000 7f858e85c640 1 -- 172.21.15.110:0/3389900528 shutdown_connections 2024-03-25T19:41:59.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.058+0000 7f858e85c640 1 -- 172.21.15.110:0/3389900528 >> 172.21.15.110:0/3389900528 conn(0x7f85880b7ea0 msgr2=0x7f85880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:59.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.058+0000 7f858e85c640 1 -- 172.21.15.110:0/3389900528 shutdown_connections 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.058+0000 7f858e85c640 1 -- 172.21.15.110:0/3389900528 wait complete. 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.059+0000 7f858e85c640 10 monclient: build_initial_monmap 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.059+0000 7f858e85c640 10 monclient: monmap: 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.059+0000 7f858e85c640 1 librados: starting msgr at 2024-03-25T19:41:59.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.059+0000 7f858e85c640 1 librados: starting objecter 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.059+0000 7f858e85c640 1 Processor -- start 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.060+0000 7f858e85c640 1 -- start start 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.060+0000 7f858e85c640 1 librados: setting wanted keys 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.060+0000 7f858e85c640 1 librados: calling monclient init 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.060+0000 7f858e85c640 10 monclient: init 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.060+0000 7f858e85c640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.060+0000 7f858e85c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.060+0000 7f858e85c640 10 monclient(hunting): picked mon.noname-c con 0x7f85880a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.060+0000 7f858e85c640 10 monclient(hunting): picked mon.noname-b con 0x7f858813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.061+0000 7f858e85c640 10 monclient(hunting): picked mon.noname-a con 0x7f85881430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.061+0000 7f858e85c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858813c9b0 con 0x7f85881430c0 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.061+0000 7f858e85c640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85881397c0 con 0x7f858813f710 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.061+0000 7f858e85c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8588139950 con 0x7f85880a98f0 2024-03-25T19:41:59.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.061+0000 7f858e85c640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:59.063 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.061+0000 7f858e85c640 10 monclient(hunting): authenticate will time out at 4290.201983s 2024-03-25T19:41:59.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.067+0000 7f858cdf9640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f85881430c0 0x7f85881e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55094/0 (socket says 172.21.15.110:55094) 2024-03-25T19:41:59.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.067+0000 7f858cdf9640 1 -- 172.21.15.110:0/241591989 learned_addr learned my addr 172.21.15.110:0/241591989 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:41:59.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.068+0000 7f8584ff9640 10 client.?.objecter ms_handle_connect 0x7f85881430c0 2024-03-25T19:41:59.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.068+0000 7f8584ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:59.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.072+0000 7f8584ff9640 10 client.?.objecter ms_handle_connect 0x7f85880a98f0 2024-03-25T19:41:59.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.072+0000 7f8584ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:59.072 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.073+0000 7f8584ff9640 10 client.?.objecter ms_handle_connect 0x7f858813f710 2024-03-25T19:41:59.072 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.073+0000 7f8584ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:59.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.077+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399230635 0 0) 0x7f858813c9b0 con 0x7f85881430c0 2024-03-25T19:41:59.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.077+0000 7f8584ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:59.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.077+0000 7f8584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:59.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.077+0000 7f8584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:59.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.077+0000 7f8584ff9640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:41:59.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.077+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8574004eb0 con 0x7f85881430c0 2024-03-25T19:41:59.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.084+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078850672 0 0) 0x7f85881397c0 con 0x7f858813f710 2024-03-25T19:41:59.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.084+0000 7f8584ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:59.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.084+0000 7f8584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:59.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.084+0000 7f8584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:59.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.084+0000 7f8584ff9640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:41:59.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.084+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f858813c9b0 con 0x7f858813f710 2024-03-25T19:41:59.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.086+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608345178 0 0) 0x7f8588139950 con 0x7f85880a98f0 2024-03-25T19:41:59.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.086+0000 7f8584ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:59.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.086+0000 7f8584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:59.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.086+0000 7f8584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:59.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.086+0000 7f8584ff9640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:41:59.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.086+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85881397c0 con 0x7f85880a98f0 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.090+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2136274588 0 0) 0x7f8574004eb0 con 0x7f85881430c0 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.090+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8588139950 con 0x7f85881430c0 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.090+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8580001f80 con 0x7f85881430c0 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.090+0000 7f8584ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.090+0000 7f8584ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.090+0000 7f8584ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:59.090 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.091 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.091 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:59.091 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:59.091 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:59.091 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:59.091 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:59.091 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:59.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.090+0000 7f8584ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:59.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.100+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1129477886 0 0) 0x7f858813c9b0 con 0x7f858813f710 2024-03-25T19:41:59.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.100+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8574004eb0 con 0x7f858813f710 2024-03-25T19:41:59.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.100+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f857c001930 con 0x7f858813f710 2024-03-25T19:41:59.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.100+0000 7f8584ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.100+0000 7f8584ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.100+0000 7f8584ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:59.100 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2747431466 0 0) 0x7f8588139950 con 0x7f85881430c0 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 >> v1:172.21.15.110:6790/0 conn(0x7f85880a98f0 legacy=0x7f8588130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 >> v1:172.21.15.178:6789/0 conn(0x7f858813f710 legacy=0x7f85881dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 1 monclient: found mon.a 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588131680 con 0x7f85881430c0 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:59.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f8584ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 5 monclient: authenticate success, global_id 4290 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 10 monclient: _renew_subs 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8588131950 con 0x7f85881430c0 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 10 monclient: _renew_subs 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8588135a30 con 0x7f85881430c0 2024-03-25T19:41:59.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.103+0000 7f858e85c640 1 librados: init done 2024-03-25T19:41:59.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.104+0000 7f858e85c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:59.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.104+0000 7f858e85c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:59.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.104+0000 7f858e85c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:59.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.104+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858813c810 con 0x7f85881430c0 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.410+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8580003c20 con 0x7f85881430c0 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.410+0000 7f8584ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.410+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8580003e50 con 0x7f85881430c0 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.410+0000 7f8584ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.410+0000 7f8584ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.410+0000 7f8584ff9640 10 monclient: dump: 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.410 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.410+0000 7f85867fc640 4 set_mon_vals no callback set 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.411+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f8580004070 con 0x7f85881430c0 2024-03-25T19:41:59.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.411+0000 7f8587fff640 10 monclient: get_auth_request con 0x7f857406df70 auth_method 0 2024-03-25T19:41:59.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.412+0000 7f8584ff9640 10 client.4290.objecter ms_handle_connect 0x7f857406df70 2024-03-25T19:41:59.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.416+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f85800549d0 con 0x7f85881430c0 2024-03-25T19:41:59.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.416+0000 7f8584ff9640 10 client.4290.objecter ms_dispatch 0x7f8588000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:59.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.416+0000 7f8584ff9640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:59.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.416+0000 7f8584ff9640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:59.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.416+0000 7f8584ff9640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T19:41:59.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.416+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85800504e0 con 0x7f85881430c0 2024-03-25T19:41:59.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.416+0000 7f8584ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:59.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.416+0000 7f8584ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:59.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.631+0000 7f858e85c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.631+0000 7f858e85c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.631+0000 7f858e85c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:41:59.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.631+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8588136fc0 con 0x7f85881430c0 2024-03-25T19:41:59.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.670+0000 7f8584ff9640 1 -- 172.21.15.110:0/241591989 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f8580052410 con 0x7f85881430c0 2024-03-25T19:41:59.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.670+0000 7f8584ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.670+0000 7f8584ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:59.670 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:41:59.670 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:59.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.672+0000 7f858e85c640 10 librados: watch_flush enter 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.672+0000 7f858e85c640 10 librados: watch_flush exit 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.672+0000 7f858e85c640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.672+0000 7f858e85c640 10 client.4290.objecter successfully canceled tick 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 >> v1:172.21.15.178:6800/1132739058 conn(0x7f857406df70 legacy=0x7f8574070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 10 monclient: shutdown 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 >> v1:172.21.15.110:6789/0 conn(0x7f85881430c0 legacy=0x7f85881e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 shutdown_connections 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 >> 172.21.15.110:0/241591989 conn(0x7f85880b7ea0 msgr2=0x7f858814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 shutdown_connections 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 1 -- 172.21.15.110:0/241591989 wait complete. 2024-03-25T19:41:59.673 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:41:59.673+0000 7f858e85c640 1 librados: shutdown 2024-03-25T19:41:59.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:06.490 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.606+0000 7ff308b72640 10 monclient: get_monmap_and_config 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.606+0000 7ff308b72640 10 monclient: build_initial_monmap 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.606+0000 7ff308b72640 10 monclient: monmap: 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:42:06.607353+0000 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:42:06.607353+0000 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:42:06.606 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:06.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.607+0000 7ff308b72640 1 Processor -- start 2024-03-25T19:42:06.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.607+0000 7ff308b72640 1 -- start start 2024-03-25T19:42:06.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.607+0000 7ff308b72640 10 monclient: init 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 10 monclient(hunting): picked mon.noname-a con 0x7ff30413fc60 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 10 monclient(hunting): picked mon.noname-c con 0x7ff304070570 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 10 monclient(hunting): picked mon.noname-b con 0x7ff3040727a0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30413c9f0 con 0x7ff30413fc60 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30406fac0 con 0x7ff304070570 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3040715d0 con 0x7ff3040727a0 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:06.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.608+0000 7ff308b72640 10 monclient(hunting): authenticate will time out at 4297.749419s 2024-03-25T19:42:06.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.614+0000 7ff30259c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff30413fc60 0x7ff30406f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60382/0 (socket says 172.21.15.110:60382) 2024-03-25T19:42:06.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.614+0000 7ff30259c640 1 -- 172.21.15.110:0/1874087389 learned_addr learned my addr 172.21.15.110:0/1874087389 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:06.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.625+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027032745 0 0) 0x7ff30413c9f0 con 0x7ff30413fc60 2024-03-25T19:42:06.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.625+0000 7ff30159a640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.625+0000 7ff30159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.625+0000 7ff30159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.625+0000 7ff30159a640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:42:06.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.625+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ec003520 con 0x7ff30413fc60 2024-03-25T19:42:06.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.634+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253698683 0 0) 0x7ff30406fac0 con 0x7ff304070570 2024-03-25T19:42:06.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.634+0000 7ff30159a640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.634+0000 7ff30159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.634+0000 7ff30159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.634+0000 7ff30159a640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:42:06.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.634+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff30413c9f0 con 0x7ff304070570 2024-03-25T19:42:06.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.636+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207527540 0 0) 0x7ff3040715d0 con 0x7ff3040727a0 2024-03-25T19:42:06.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.636+0000 7ff30159a640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.636+0000 7ff30159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.636+0000 7ff30159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.636+0000 7ff30159a640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:42:06.636 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.637+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff30406fac0 con 0x7ff3040727a0 2024-03-25T19:42:06.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377554208 0 0) 0x7ff30413c9f0 con 0x7ff304070570 2024-03-25T19:42:06.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 >> v1:172.21.15.110:6790/0 conn(0x7ff3040727a0 legacy=0x7ff30414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 >> v1:172.21.15.110:6789/0 conn(0x7ff30413fc60 legacy=0x7ff30406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 10 monclient: _finish_hunting 0 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 1 monclient: found mon.noname-c 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3040718b0 con 0x7ff304070570 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 10 monclient: _finish_auth 0 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 10 monclient: _check_auth_tickets 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff2f8001670 con 0x7ff304070570 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff30159a640 10 monclient: dump: 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:06.648 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff308b72640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:42:06.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.648+0000 7ff308b72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:06.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2f80017d0 con 0x7ff304070570 2024-03-25T19:42:06.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff30159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:06.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff30159a640 1 -- 172.21.15.110:0/1874087389 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff2f8001be0 con 0x7ff304070570 2024-03-25T19:42:06.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff308b72640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:06.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff308b72640 4 set_mon_vals no callback set 2024-03-25T19:42:06.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff30159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff30159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff30159a640 10 monclient: dump: 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:06.665 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff308b72640 10 monclient: shutdown 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff308b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.665+0000 7ff308b72640 1 -- 172.21.15.110:0/1874087389 >> v1:172.21.15.178:6789/0 conn(0x7ff304070570 legacy=0x7ff304072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.666+0000 7ff308b72640 1 -- 172.21.15.110:0/1874087389 shutdown_connections 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.666+0000 7ff308b72640 1 -- 172.21.15.110:0/1874087389 >> 172.21.15.110:0/1874087389 conn(0x7ff3040b4a80 msgr2=0x7ff3040b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.666+0000 7ff308b72640 1 -- 172.21.15.110:0/1874087389 shutdown_connections 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.666+0000 7ff308b72640 1 -- 172.21.15.110:0/1874087389 wait complete. 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 10 monclient: build_initial_monmap 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 10 monclient: monmap: 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:06.667 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 1 librados: starting msgr at 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 1 librados: starting objecter 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 1 Processor -- start 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 1 -- start start 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 1 librados: setting wanted keys 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 1 librados: calling monclient init 2024-03-25T19:42:06.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.667+0000 7ff308b72640 10 monclient: init 2024-03-25T19:42:06.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:06.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:06.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 10 monclient(hunting): picked mon.noname-c con 0x7ff304070570 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:06.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 10 monclient(hunting): picked mon.noname-a con 0x7ff3040727a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:06.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 10 monclient(hunting): picked mon.noname-b con 0x7ff30413fc60 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:06.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3041cb000 con 0x7ff3040727a0 2024-03-25T19:42:06.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3041cc7e0 con 0x7ff30413fc60 2024-03-25T19:42:06.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3041db210 con 0x7ff304070570 2024-03-25T19:42:06.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:06.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.668+0000 7ff308b72640 10 monclient(hunting): authenticate will time out at 4297.809046s 2024-03-25T19:42:06.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.677+0000 7ff302d9d640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff30413fc60 0x7ff3041e20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:33016/0 (socket says 172.21.15.110:33016) 2024-03-25T19:42:06.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.677+0000 7ff302d9d640 1 -- 172.21.15.110:0/1524524214 learned_addr learned my addr 172.21.15.110:0/1524524214 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:06.678 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.679+0000 7ff2f2ffd640 10 client.?.objecter ms_handle_connect 0x7ff30413fc60 2024-03-25T19:42:06.678 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.679+0000 7ff2f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.683+0000 7ff2f2ffd640 10 client.?.objecter ms_handle_connect 0x7ff304070570 2024-03-25T19:42:06.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.683+0000 7ff2f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.690+0000 7ff2f2ffd640 10 client.?.objecter ms_handle_connect 0x7ff3040727a0 2024-03-25T19:42:06.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.690+0000 7ff2f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.690+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110659486 0 0) 0x7ff3041cc7e0 con 0x7ff30413fc60 2024-03-25T19:42:06.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.690+0000 7ff2f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.690+0000 7ff2f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.690+0000 7ff2f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.690+0000 7ff2f2ffd640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:42:06.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.690+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e4004eb0 con 0x7ff30413fc60 2024-03-25T19:42:06.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.700+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055277935 0 0) 0x7ff3041cb000 con 0x7ff3040727a0 2024-03-25T19:42:06.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.700+0000 7ff2f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.700+0000 7ff2f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.700+0000 7ff2f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.700+0000 7ff2f2ffd640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:42:06.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.700+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3041cc7e0 con 0x7ff3040727a0 2024-03-25T19:42:06.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.703+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858150316 0 0) 0x7ff3041db210 con 0x7ff304070570 2024-03-25T19:42:06.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.703+0000 7ff2f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.703+0000 7ff2f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.703+0000 7ff2f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.703+0000 7ff2f2ffd640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:42:06.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.703+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3041cb000 con 0x7ff304070570 2024-03-25T19:42:06.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.712+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4011331899 0 0) 0x7ff2e4004eb0 con 0x7ff30413fc60 2024-03-25T19:42:06.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.712+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3041db210 con 0x7ff30413fc60 2024-03-25T19:42:06.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.712+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff2fc001cd0 con 0x7ff30413fc60 2024-03-25T19:42:06.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.712+0000 7ff2f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.712+0000 7ff2f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.712+0000 7ff2f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:06.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.712+0000 7ff2f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.713+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031091118 0 0) 0x7ff3041cc7e0 con 0x7ff3040727a0 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.713+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2e4004eb0 con 0x7ff3040727a0 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.713+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff2f8001ae0 con 0x7ff3040727a0 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.713+0000 7ff2f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.713+0000 7ff2f2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.713+0000 7ff2f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:06.713 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:06.714 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.714 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:06.714 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:06.714 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:06.714 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:06.714 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3484801099 0 0) 0x7ff2e4004eb0 con 0x7ff3040727a0 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 >> v1:172.21.15.110:6790/0 conn(0x7ff304070570 legacy=0x7ff3041ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 >> v1:172.21.15.178:6789/0 conn(0x7ff30413fc60 legacy=0x7ff3041e20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 1 monclient: found mon.a 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3041db3a0 con 0x7ff3040727a0 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff2f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 5 monclient: authenticate success, global_id 4296 2024-03-25T19:42:06.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 10 monclient: _renew_subs 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3041cb490 con 0x7ff3040727a0 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 10 monclient: _renew_subs 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff304142d10 con 0x7ff3040727a0 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.723+0000 7ff308b72640 1 librados: init done 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.724+0000 7ff308b72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.724+0000 7ff308b72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.724+0000 7ff308b72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:06.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:06.724+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3041cc5e0 con 0x7ff3040727a0 2024-03-25T19:42:07.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.035+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2f8001210 con 0x7ff3040727a0 2024-03-25T19:42:07.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.035+0000 7ff2f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:07.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.035+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff2f8003a40 con 0x7ff3040727a0 2024-03-25T19:42:07.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.035+0000 7ff2f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.035+0000 7ff2f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:07.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.035+0000 7ff2f2ffd640 10 monclient: dump: 2024-03-25T19:42:07.035 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:07.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.035+0000 7ff300d99640 4 set_mon_vals no callback set 2024-03-25T19:42:07.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.037+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7ff2f8019a10 con 0x7ff3040727a0 2024-03-25T19:42:07.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.037+0000 7ff30259c640 10 monclient: get_auth_request con 0x7ff2e406dfd0 auth_method 0 2024-03-25T19:42:07.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.038+0000 7ff2f2ffd640 10 client.4296.objecter ms_handle_connect 0x7ff2e406dfd0 2024-03-25T19:42:07.042 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.043+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7ff2f80536a0 con 0x7ff3040727a0 2024-03-25T19:42:07.042 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.043+0000 7ff2f2ffd640 10 client.4296.objecter ms_dispatch 0x7ff304000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:07.042 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.043+0000 7ff2f2ffd640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:07.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.043+0000 7ff2f2ffd640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:07.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.043+0000 7ff2f2ffd640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-25T19:42:07.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.043+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2f804efe0 con 0x7ff3040727a0 2024-03-25T19:42:07.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.043+0000 7ff2f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:07.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.043+0000 7ff2f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:07.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.287+0000 7ff308b72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:07.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.287+0000 7ff308b72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:07.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.287+0000 7ff308b72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:07.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.287+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3041cb1b0 con 0x7ff3040727a0 2024-03-25T19:42:07.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.316+0000 7ff2f2ffd640 1 -- 172.21.15.110:0/1524524214 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7ff3041cb1b0 con 0x7ff3040727a0 2024-03-25T19:42:07.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.316+0000 7ff2f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:07.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.316+0000 7ff2f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:07.316 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:42:07.316 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:07.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 10 librados: watch_flush enter 2024-03-25T19:42:07.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 10 librados: watch_flush exit 2024-03-25T19:42:07.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-25T19:42:07.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 10 client.4296.objecter successfully canceled tick 2024-03-25T19:42:07.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 >> v1:172.21.15.178:6800/1132739058 conn(0x7ff2e406dfd0 legacy=0x7ff2e4070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 10 monclient: shutdown 2024-03-25T19:42:07.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:07.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 >> v1:172.21.15.110:6789/0 conn(0x7ff3040727a0 legacy=0x7ff3041dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 shutdown_connections 2024-03-25T19:42:07.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 >> 172.21.15.110:0/1524524214 conn(0x7ff3040b4a80 msgr2=0x7ff3040ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:07.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 shutdown_connections 2024-03-25T19:42:07.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 1 -- 172.21.15.110:0/1524524214 wait complete. 2024-03-25T19:42:07.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:07.319+0000 7ff308b72640 1 librados: shutdown 2024-03-25T19:42:07.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:14.138 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:14.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.252+0000 7f5834932640 10 monclient: get_monmap_and_config 2024-03-25T19:42:14.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.252+0000 7f5834932640 10 monclient: build_initial_monmap 2024-03-25T19:42:14.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.252+0000 7f5834932640 10 monclient: monmap: 2024-03-25T19:42:14.252 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:42:14.253877+0000 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:42:14.253877+0000 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:42:14.253 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:14.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.253+0000 7f5834932640 1 Processor -- start 2024-03-25T19:42:14.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 1 -- start start 2024-03-25T19:42:14.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 10 monclient: init 2024-03-25T19:42:14.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:14.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:14.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 10 monclient(hunting): picked mon.noname-a con 0x7f58300a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:14.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 10 monclient(hunting): picked mon.noname-c con 0x7f583013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:14.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 10 monclient(hunting): picked mon.noname-b con 0x7f58301430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:14.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5830142520 con 0x7f58300a98f0 2024-03-25T19:42:14.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58300b28a0 con 0x7f583013f710 2024-03-25T19:42:14.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58300b2a30 con 0x7f58301430c0 2024-03-25T19:42:14.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:14.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.254+0000 7f5834932640 10 monclient(hunting): authenticate will time out at 4305.395535s 2024-03-25T19:42:14.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.262+0000 7f5827fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f583013f710 0x7f58301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:60252/0 (socket says 172.21.15.110:60252) 2024-03-25T19:42:14.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.262+0000 7f5827fff640 1 -- 172.21.15.110:0/1427424237 learned_addr learned my addr 172.21.15.110:0/1427424237 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:14.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.273+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679707873 0 0) 0x7f58300b28a0 con 0x7f583013f710 2024-03-25T19:42:14.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.273+0000 7f582dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.273+0000 7f582dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.273+0000 7f582dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.273+0000 7f582dd9b640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:42:14.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.273+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f581c003520 con 0x7f583013f710 2024-03-25T19:42:14.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.280+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372204533 0 0) 0x7f58300b2a30 con 0x7f58301430c0 2024-03-25T19:42:14.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.280+0000 7f582dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.280+0000 7f582dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.280+0000 7f582dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.280+0000 7f582dd9b640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:42:14.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.280+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58300b28a0 con 0x7f58301430c0 2024-03-25T19:42:14.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1581815412 0 0) 0x7f581c003520 con 0x7f583013f710 2024-03-25T19:42:14.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 >> v1:172.21.15.110:6790/0 conn(0x7f58301430c0 legacy=0x7f583014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 >> v1:172.21.15.110:6789/0 conn(0x7f58300a98f0 legacy=0x7f58300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 1 monclient: found mon.noname-c 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58300b44c0 con 0x7f583013f710 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f582dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f5834932640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:42:14.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.285+0000 7f5834932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.286+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5818001650 con 0x7f583013f710 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.286+0000 7f582dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.286+0000 7f582dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.286+0000 7f582dd9b640 10 monclient: dump: 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.286 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:14.287 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:14.287 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:14.287 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:14.287 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:14.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.286+0000 7f5834932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:14.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.301+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58180018a0 con 0x7f583013f710 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.301+0000 7f582dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.301+0000 7f582dd9b640 1 -- 172.21.15.110:0/1427424237 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5818001c30 con 0x7f583013f710 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.301+0000 7f582dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.301+0000 7f582dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.301+0000 7f582dd9b640 10 monclient: dump: 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:14.301 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:14.302 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:14.302 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:14.302 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:14.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.301+0000 7f5834932640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:14.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.301+0000 7f5834932640 4 set_mon_vals no callback set 2024-03-25T19:42:14.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.302+0000 7f5834932640 10 monclient: shutdown 2024-03-25T19:42:14.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.302+0000 7f5834932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:14.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.302+0000 7f5834932640 1 -- 172.21.15.110:0/1427424237 >> v1:172.21.15.178:6789/0 conn(0x7f583013f710 legacy=0x7f58301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.302+0000 7f5834932640 1 -- 172.21.15.110:0/1427424237 shutdown_connections 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.302+0000 7f5834932640 1 -- 172.21.15.110:0/1427424237 >> 172.21.15.110:0/1427424237 conn(0x7f58300b7ea0 msgr2=0x7f58300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.302+0000 7f5834932640 1 -- 172.21.15.110:0/1427424237 shutdown_connections 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.302+0000 7f5834932640 1 -- 172.21.15.110:0/1427424237 wait complete. 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.303+0000 7f5834932640 10 monclient: build_initial_monmap 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.303+0000 7f5834932640 10 monclient: monmap: 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:42:14.304 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.303+0000 7f5834932640 1 librados: starting msgr at 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.303+0000 7f5834932640 1 librados: starting objecter 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.304+0000 7f5834932640 1 Processor -- start 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.304+0000 7f5834932640 1 -- start start 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.304+0000 7f5834932640 1 librados: setting wanted keys 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.304+0000 7f5834932640 1 librados: calling monclient init 2024-03-25T19:42:14.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.304+0000 7f5834932640 10 monclient: init 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.304+0000 7f5834932640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.304+0000 7f5834932640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.305+0000 7f5834932640 10 monclient(hunting): picked mon.noname-c con 0x7f58300a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.305+0000 7f5834932640 10 monclient(hunting): picked mon.noname-b con 0x7f583013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.305+0000 7f5834932640 10 monclient(hunting): picked mon.noname-a con 0x7f58301430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.305+0000 7f5834932640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5830138d50 con 0x7f58301430c0 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.305+0000 7f5834932640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5830132e80 con 0x7f583013f710 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.305+0000 7f5834932640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5830139030 con 0x7f58300a98f0 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.305+0000 7f5834932640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:14.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.305+0000 7f5834932640 10 monclient(hunting): authenticate will time out at 4305.446103s 2024-03-25T19:42:14.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.313+0000 7f582ed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f58301430c0 0x7f5830132770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:49898/0 (socket says 172.21.15.110:49898) 2024-03-25T19:42:14.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.313+0000 7f582ed9d640 1 -- 172.21.15.110:0/2567801874 learned_addr learned my addr 172.21.15.110:0/2567801874 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:14.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.314+0000 7f5826ffd640 10 client.?.objecter ms_handle_connect 0x7f58300a98f0 2024-03-25T19:42:14.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.314+0000 7f5826ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:14.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.320+0000 7f5826ffd640 10 client.?.objecter ms_handle_connect 0x7f58301430c0 2024-03-25T19:42:14.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.320+0000 7f5826ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:14.322 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.322+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664857109 0 0) 0x7f5830139030 con 0x7f58300a98f0 2024-03-25T19:42:14.322 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.322+0000 7f5826ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.322 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.322+0000 7f5826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.322 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.322+0000 7f5826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.322 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.322+0000 7f5826ffd640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:42:14.322 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.322+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5810004eb0 con 0x7f58300a98f0 2024-03-25T19:42:14.322 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.322+0000 7f5826ffd640 10 client.?.objecter ms_handle_connect 0x7f583013f710 2024-03-25T19:42:14.322 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.322+0000 7f5826ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:14.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.324+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356444936 0 0) 0x7f5830138d50 con 0x7f58301430c0 2024-03-25T19:42:14.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.324+0000 7f5826ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.324+0000 7f5826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.324+0000 7f5826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.324+0000 7f5826ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:42:14.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.324+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5830139030 con 0x7f58301430c0 2024-03-25T19:42:14.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.329+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803336163 0 0) 0x7f5830132e80 con 0x7f583013f710 2024-03-25T19:42:14.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.329+0000 7f5826ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.329+0000 7f5826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.329+0000 7f5826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.329+0000 7f5826ffd640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:42:14.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.329+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5830138d50 con 0x7f583013f710 2024-03-25T19:42:14.333 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.333+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2439784028 0 0) 0x7f5830139030 con 0x7f58301430c0 2024-03-25T19:42:14.333 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.333+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5830132e80 con 0x7f58301430c0 2024-03-25T19:42:14.333 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.334+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5828001c10 con 0x7f58301430c0 2024-03-25T19:42:14.333 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.334+0000 7f5826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.333 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.334+0000 7f5826ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:14.333 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.334+0000 7f5826ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:14.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.334+0000 7f5826ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:14.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.341+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2259795640 0 0) 0x7f5810004eb0 con 0x7f58300a98f0 2024-03-25T19:42:14.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.341+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5830139030 con 0x7f58300a98f0 2024-03-25T19:42:14.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.341+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58200018c0 con 0x7f58300a98f0 2024-03-25T19:42:14.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.341+0000 7f5826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.341+0000 7f5826ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.341+0000 7f5826ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:14.341 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1135304560 0 0) 0x7f5830132e80 con 0x7f58301430c0 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 >> v1:172.21.15.110:6790/0 conn(0x7f58300a98f0 legacy=0x7f58300ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 >> v1:172.21.15.178:6789/0 conn(0x7f583013f710 legacy=0x7f5830132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 1 monclient: found mon.a 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830139310 con 0x7f58301430c0 2024-03-25T19:42:14.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 5 monclient: authenticate success, global_id 4302 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 10 monclient: _renew_subs 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5830137630 con 0x7f58301430c0 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 10 monclient: _renew_subs 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5830138390 con 0x7f58301430c0 2024-03-25T19:42:14.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.345+0000 7f5834932640 1 librados: init done 2024-03-25T19:42:14.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.346+0000 7f5834932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.346+0000 7f5834932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.346+0000 7f5834932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:14.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.346+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58300627b0 con 0x7f58301430c0 2024-03-25T19:42:14.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.660+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5828003990 con 0x7f58301430c0 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.660+0000 7f5826ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.660+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5828002100 con 0x7f58301430c0 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.660+0000 7f5826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.660+0000 7f5826ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.660+0000 7f5826ffd640 10 monclient: dump: 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:14.660 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:14.661 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:14.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.660+0000 7f582d59a640 4 set_mon_vals no callback set 2024-03-25T19:42:14.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.660+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f5828003de0 con 0x7f58301430c0 2024-03-25T19:42:14.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.661+0000 7f582e59c640 10 monclient: get_auth_request con 0x7f581006df70 auth_method 0 2024-03-25T19:42:14.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.661+0000 7f5826ffd640 10 client.4302.objecter ms_handle_connect 0x7f581006df70 2024-03-25T19:42:14.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.666+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f58280047e0 con 0x7f58301430c0 2024-03-25T19:42:14.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.666+0000 7f5826ffd640 10 client.4302.objecter ms_dispatch 0x7f5830000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:14.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.666+0000 7f5826ffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:14.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.666+0000 7f5826ffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:14.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.666+0000 7f5826ffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T19:42:14.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.667+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5828050560 con 0x7f58301430c0 2024-03-25T19:42:14.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.667+0000 7f5826ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.667+0000 7f5826ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:14.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.915+0000 7f5834932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.915+0000 7f5834932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.915+0000 7f5834932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:14.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.915+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f583013dc00 con 0x7f58301430c0 2024-03-25T19:42:14.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.943+0000 7f5826ffd640 1 -- 172.21.15.110:0/2567801874 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f5828052490 con 0x7f58301430c0 2024-03-25T19:42:14.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.943+0000 7f5826ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.943+0000 7f5826ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:14.943 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:42:14.943 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:14.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.945+0000 7f5834932640 10 librados: watch_flush enter 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.945+0000 7f5834932640 10 librados: watch_flush exit 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.945+0000 7f5834932640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.945+0000 7f5834932640 10 client.4302.objecter successfully canceled tick 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.945+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 >> v1:172.21.15.178:6800/1132739058 conn(0x7f581006df70 legacy=0x7f5810070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.946+0000 7f5834932640 10 monclient: shutdown 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.946+0000 7f5834932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.946+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 >> v1:172.21.15.110:6789/0 conn(0x7f58301430c0 legacy=0x7f5830132770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.946+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 shutdown_connections 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.946+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 >> 172.21.15.110:0/2567801874 conn(0x7f58300b7ea0 msgr2=0x7f5830074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.946+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 shutdown_connections 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.946+0000 7f5834932640 1 -- 172.21.15.110:0/2567801874 wait complete. 2024-03-25T19:42:14.946 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:14.946+0000 7f5834932640 1 librados: shutdown 2024-03-25T19:42:14.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:17.163 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:24.447 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:17.176 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:24.460 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:17.212 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:24.496 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:17.213 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:24.468 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:17.224 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:24.479 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:17.253 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:24.538 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:17.259 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:24.514 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:17.261 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:24.546 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:17.288 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:24.573 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:21.763 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.871+0000 7f6d56a2a640 10 monclient: get_monmap_and_config 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.871+0000 7f6d56a2a640 10 monclient: build_initial_monmap 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.871+0000 7f6d56a2a640 10 monclient: monmap: 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:42:21.872542+0000 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:42:21.872542+0000 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:21.871 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:21.872 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:21.872 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:42:21.872 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:42:21.872 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:21.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.872+0000 7f6d56a2a640 1 Processor -- start 2024-03-25T19:42:21.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.872+0000 7f6d56a2a640 1 -- start start 2024-03-25T19:42:21.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.872+0000 7f6d56a2a640 10 monclient: init 2024-03-25T19:42:21.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:21.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:21.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f6d500a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:21.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f6d5013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:21.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f6d501430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:21.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d50142520 con 0x7f6d5013f710 2024-03-25T19:42:21.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d500b28a0 con 0x7f6d501430c0 2024-03-25T19:42:21.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d500b2a30 con 0x7f6d500a98f0 2024-03-25T19:42:21.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:21.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.873+0000 7f6d56a2a640 10 monclient(hunting): authenticate will time out at 4313.014444s 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.899+0000 7f6d54fc7640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f6d501430c0 0x7f6d5014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:57210/0 (socket says 172.21.15.110:57210) 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.899+0000 7f6d54fc7640 1 -- 172.21.15.110:0/2831339821 learned_addr learned my addr 172.21.15.110:0/2831339821 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.911+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137006407 0 0) 0x7f6d500b2a30 con 0x7f6d500a98f0 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.911+0000 7f6d4effd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.911+0000 7f6d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.911+0000 7f6d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.911+0000 7f6d4effd640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.911+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d38003520 con 0x7f6d500a98f0 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.913+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801150323 0 0) 0x7f6d500b28a0 con 0x7f6d501430c0 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.913+0000 7f6d4effd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.913+0000 7f6d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.913+0000 7f6d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.913+0000 7f6d4effd640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:42:21.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.913+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d500b2a30 con 0x7f6d501430c0 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1242454827 0 0) 0x7f6d500b2a30 con 0x7f6d501430c0 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 >> v1:172.21.15.110:6790/0 conn(0x7f6d500a98f0 legacy=0x7f6d500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 >> v1:172.21.15.110:6789/0 conn(0x7f6d5013f710 legacy=0x7f6d501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 1 monclient: found mon.noname-c 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500b44c0 con 0x7f6d501430c0 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 10 monclient: _finish_auth 0 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6d480019f0 con 0x7f6d501430c0 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d56a2a640 5 monclient: authenticate success, global_id 4315 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d56a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d4effd640 10 monclient: dump: 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:21.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.919+0000 7f6d56a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:21.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d48001b50 con 0x7f6d501430c0 2024-03-25T19:42:21.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:21.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d4effd640 1 -- 172.21.15.110:0/2831339821 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6d48001f20 con 0x7f6d501430c0 2024-03-25T19:42:21.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d4effd640 10 monclient: dump: 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:21.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d56a2a640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:21.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d56a2a640 4 set_mon_vals no callback set 2024-03-25T19:42:21.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.936+0000 7f6d56a2a640 10 monclient: shutdown 2024-03-25T19:42:21.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.937+0000 7f6d56a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:21.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.937+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2831339821 >> v1:172.21.15.178:6789/0 conn(0x7f6d501430c0 legacy=0x7f6d5014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.937+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2831339821 shutdown_connections 2024-03-25T19:42:21.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.937+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2831339821 >> 172.21.15.110:0/2831339821 conn(0x7f6d500b7ea0 msgr2=0x7f6d500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:21.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.937+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2831339821 shutdown_connections 2024-03-25T19:42:21.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.937+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2831339821 wait complete. 2024-03-25T19:42:21.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.937+0000 7f6d56a2a640 10 monclient: build_initial_monmap 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.937+0000 7f6d56a2a640 10 monclient: monmap: 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.938+0000 7f6d56a2a640 1 librados: starting msgr at 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.938+0000 7f6d56a2a640 1 librados: starting objecter 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.938+0000 7f6d56a2a640 1 Processor -- start 2024-03-25T19:42:21.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.938+0000 7f6d56a2a640 1 -- start start 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.938+0000 7f6d56a2a640 1 librados: setting wanted keys 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.938+0000 7f6d56a2a640 1 librados: calling monclient init 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.938+0000 7f6d56a2a640 10 monclient: init 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f6d500a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f6d5013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f6d501430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d50071610 con 0x7f6d500a98f0 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d50070370 con 0x7f6d5013f710 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d50071210 con 0x7f6d501430c0 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:21.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.939+0000 7f6d56a2a640 10 monclient(hunting): authenticate will time out at 4313.080262s 2024-03-25T19:42:21.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.944+0000 7f6d4ffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6d500a98f0 0x7f6d50070b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58820/0 (socket says 172.21.15.110:58820) 2024-03-25T19:42:21.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.944+0000 7f6d4ffff640 1 -- 172.21.15.110:0/2912530866 learned_addr learned my addr 172.21.15.110:0/2912530866 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:21.948 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.949+0000 7f6d4cff9640 10 client.?.objecter ms_handle_connect 0x7f6d500a98f0 2024-03-25T19:42:21.948 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.949+0000 7f6d4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.953+0000 7f6d4cff9640 10 client.?.objecter ms_handle_connect 0x7f6d5013f710 2024-03-25T19:42:21.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.953+0000 7f6d4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.962+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770434449 0 0) 0x7f6d50071610 con 0x7f6d500a98f0 2024-03-25T19:42:21.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.962+0000 7f6d4cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.962+0000 7f6d4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.962+0000 7f6d4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.962+0000 7f6d4cff9640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:42:21.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.962+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d3c004eb0 con 0x7f6d500a98f0 2024-03-25T19:42:21.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.964+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349524286 0 0) 0x7f6d50070370 con 0x7f6d5013f710 2024-03-25T19:42:21.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.964+0000 7f6d4cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.964+0000 7f6d4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.964+0000 7f6d4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.964+0000 7f6d4cff9640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:42:21.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.964+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d50071610 con 0x7f6d5013f710 2024-03-25T19:42:21.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.967+0000 7f6d4cff9640 10 client.?.objecter ms_handle_connect 0x7f6d501430c0 2024-03-25T19:42:21.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.967+0000 7f6d4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.975+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2800971720 0 0) 0x7f6d3c004eb0 con 0x7f6d500a98f0 2024-03-25T19:42:21.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.975+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d50070370 con 0x7f6d500a98f0 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.975+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6d40001a10 con 0x7f6d500a98f0 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.975+0000 7f6d4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.975+0000 7f6d4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.975+0000 7f6d4cff9640 10 monclient(hunting): dump: 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:21.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.975+0000 7f6d4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.979+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812166621 0 0) 0x7f6d50071610 con 0x7f6d5013f710 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.979+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d3c004eb0 con 0x7f6d5013f710 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.979+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6d440018a0 con 0x7f6d5013f710 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.979+0000 7f6d4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.979+0000 7f6d4cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.979+0000 7f6d4cff9640 10 monclient(hunting): dump: 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:21.979 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:21.980 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.980 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:21.980 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:21.980 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:21.980 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:21.980 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:21.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.986+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218004323 0 0) 0x7f6d50071210 con 0x7f6d501430c0 2024-03-25T19:42:21.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.986+0000 7f6d4cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.986+0000 7f6d4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.986+0000 7f6d4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.986+0000 7f6d4cff9640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:42:21.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:21.986+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d50071610 con 0x7f6d501430c0 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 761549422 0 0) 0x7f6d3c004eb0 con 0x7f6d5013f710 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 >> v1:172.21.15.110:6790/0 conn(0x7f6d501430c0 legacy=0x7f6d501e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 >> v1:172.21.15.110:6789/0 conn(0x7f6d500a98f0 legacy=0x7f6d50070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 1 monclient: found mon.b 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d50071d90 con 0x7f6d5013f710 2024-03-25T19:42:22.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 5 monclient: authenticate success, global_id 4318 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 10 monclient: _renew_subs 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d50071fe0 con 0x7f6d5013f710 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 10 monclient: _renew_subs 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d501dc280 con 0x7f6d5013f710 2024-03-25T19:42:22.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.000+0000 7f6d56a2a640 1 librados: init done 2024-03-25T19:42:22.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.001+0000 7f6d56a2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:22.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.001+0000 7f6d56a2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:22.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.001+0000 7f6d56a2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:22.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.001+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d500627b0 con 0x7f6d5013f710 2024-03-25T19:42:22.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.288+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d440035f0 con 0x7f6d5013f710 2024-03-25T19:42:22.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.288+0000 7f6d4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:22.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.288+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6d440039c0 con 0x7f6d5013f710 2024-03-25T19:42:22.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.288+0000 7f6d4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.288+0000 7f6d4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.288+0000 7f6d4cff9640 10 monclient: dump: 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:22.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.288+0000 7f6d4e7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:22.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.289+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f6d44003be0 con 0x7f6d5013f710 2024-03-25T19:42:22.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.290+0000 7f6d4ffff640 10 monclient: get_auth_request con 0x7f6d3c06df40 auth_method 0 2024-03-25T19:42:22.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.290+0000 7f6d4cff9640 10 client.4318.objecter ms_handle_connect 0x7f6d3c06df40 2024-03-25T19:42:22.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.295+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f6d44053510 con 0x7f6d5013f710 2024-03-25T19:42:22.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.295+0000 7f6d4cff9640 10 client.4318.objecter ms_dispatch 0x7f6d50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:22.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.295+0000 7f6d4cff9640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:22.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.295+0000 7f6d4cff9640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:22.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.295+0000 7f6d4cff9640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T19:42:22.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.295+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d4404f020 con 0x7f6d5013f710 2024-03-25T19:42:22.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.295+0000 7f6d4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:22.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.295+0000 7f6d4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:22.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.540+0000 7f6d56a2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.540+0000 7f6d56a2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.540+0000 7f6d56a2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:22.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.540+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d501db6b0 con 0x7f6d5013f710 2024-03-25T19:42:22.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.561+0000 7f6d4cff9640 1 -- 172.21.15.110:0/2912530866 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f6d501db6b0 con 0x7f6d5013f710 2024-03-25T19:42:22.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.562+0000 7f6d4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.562+0000 7f6d4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:22.561 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:42:22.562 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:22.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 10 librados: watch_flush enter 2024-03-25T19:42:22.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 10 librados: watch_flush exit 2024-03-25T19:42:22.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T19:42:22.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 10 client.4318.objecter successfully canceled tick 2024-03-25T19:42:22.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 >> v1:172.21.15.178:6800/1132739058 conn(0x7f6d3c06df40 legacy=0x7f6d3c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 10 monclient: shutdown 2024-03-25T19:42:22.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:22.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 >> v1:172.21.15.178:6789/0 conn(0x7f6d5013f710 legacy=0x7f6d501db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 shutdown_connections 2024-03-25T19:42:22.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 >> 172.21.15.110:0/2912530866 conn(0x7f6d500b7ea0 msgr2=0x7f6d50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:22.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 shutdown_connections 2024-03-25T19:42:22.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 1 -- 172.21.15.110:0/2912530866 wait complete. 2024-03-25T19:42:22.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:22.564+0000 7f6d56a2a640 1 librados: shutdown 2024-03-25T19:42:22.574 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:29.381 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.494+0000 7ff4d4ba7640 10 monclient: get_monmap_and_config 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.494+0000 7ff4d4ba7640 10 monclient: build_initial_monmap 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.494+0000 7ff4d4ba7640 10 monclient: monmap: 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:42:29.495218+0000 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:42:29.495218+0000 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:29.494 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:42:29.495 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:42:29.495 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 1 Processor -- start 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 1 -- start start 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 10 monclient: init 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d00a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d013f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d01430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:29.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d0142520 con 0x7ff4d01430c0 2024-03-25T19:42:29.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d00b28a0 con 0x7ff4d00a98f0 2024-03-25T19:42:29.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d00b2a30 con 0x7ff4d013f710 2024-03-25T19:42:29.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:29.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.495+0000 7ff4d4ba7640 10 monclient(hunting): authenticate will time out at 4320.636754s 2024-03-25T19:42:29.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.501+0000 7ff4ce59c640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff4d00a98f0 0x7ff4d00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:33890/0 (socket says 172.21.15.110:33890) 2024-03-25T19:42:29.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.501+0000 7ff4ce59c640 1 -- 172.21.15.110:0/235804566 learned_addr learned my addr 172.21.15.110:0/235804566 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:29.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.518+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732404875 0 0) 0x7ff4d00b28a0 con 0x7ff4d00a98f0 2024-03-25T19:42:29.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.518+0000 7ff4cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.518+0000 7ff4cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.518+0000 7ff4cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.518+0000 7ff4cdd9b640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:42:29.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.518+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4bc003520 con 0x7ff4d00a98f0 2024-03-25T19:42:29.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.520+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058949244 0 0) 0x7ff4d0142520 con 0x7ff4d01430c0 2024-03-25T19:42:29.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.520+0000 7ff4cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.520+0000 7ff4cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.520+0000 7ff4cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.520+0000 7ff4cdd9b640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:42:29.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.520+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d00b28a0 con 0x7ff4d01430c0 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423179346 0 0) 0x7ff4d00b28a0 con 0x7ff4d01430c0 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 >> v1:172.21.15.110:6790/0 conn(0x7ff4d013f710 legacy=0x7ff4d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 >> v1:172.21.15.178:6789/0 conn(0x7ff4d00a98f0 legacy=0x7ff4d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 1 monclient: found mon.noname-a 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d00b44c0 con 0x7ff4d01430c0 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:29.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff4c80019f0 con 0x7ff4d01430c0 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4d4ba7640 5 monclient: authenticate success, global_id 4311 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4d4ba7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4cdd9b640 10 monclient: dump: 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:29.528 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:29.529 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:29.529 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.527+0000 7ff4d4ba7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:29.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.535+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4c8001b50 con 0x7ff4d01430c0 2024-03-25T19:42:29.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.535+0000 7ff4cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.535+0000 7ff4cdd9b640 1 -- 172.21.15.110:0/235804566 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff4c8001f20 con 0x7ff4d01430c0 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.535+0000 7ff4cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.535+0000 7ff4cdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.535+0000 7ff4cdd9b640 10 monclient: dump: 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:29.535 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:29.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.535+0000 7ff4d4ba7640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:29.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.535+0000 7ff4d4ba7640 4 set_mon_vals no callback set 2024-03-25T19:42:29.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.536+0000 7ff4d4ba7640 10 monclient: shutdown 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.536+0000 7ff4d4ba7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.536+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/235804566 >> v1:172.21.15.110:6789/0 conn(0x7ff4d01430c0 legacy=0x7ff4d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.536+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/235804566 shutdown_connections 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.536+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/235804566 >> 172.21.15.110:0/235804566 conn(0x7ff4d00b7ea0 msgr2=0x7ff4d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.536+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/235804566 shutdown_connections 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.536+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/235804566 wait complete. 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.537+0000 7ff4d4ba7640 10 monclient: build_initial_monmap 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.537+0000 7ff4d4ba7640 10 monclient: monmap: 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:42:29.537 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:29.538 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:42:29.538 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:42:29.538 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 librados: starting msgr at 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 librados: starting objecter 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 Processor -- start 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 -- start start 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 librados: setting wanted keys 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 librados: calling monclient init 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 10 monclient: init 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d00a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d01430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d0138a60 con 0x7ff4d00a98f0 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d01397c0 con 0x7ff4d013f710 2024-03-25T19:42:29.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d01d7810 con 0x7ff4d01430c0 2024-03-25T19:42:29.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:29.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.538+0000 7ff4d4ba7640 10 monclient(hunting): authenticate will time out at 4320.679767s 2024-03-25T19:42:29.543 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.544+0000 7ff4c7fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff4d013f710 0x7ff4d01d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:33914/0 (socket says 172.21.15.110:33914) 2024-03-25T19:42:29.543 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.544+0000 7ff4c7fff640 1 -- 172.21.15.110:0/1098302075 learned_addr learned my addr 172.21.15.110:0/1098302075 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:29.544 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.545+0000 7ff4c6ffd640 10 client.?.objecter ms_handle_connect 0x7ff4d013f710 2024-03-25T19:42:29.544 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.545+0000 7ff4c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.550+0000 7ff4c6ffd640 10 client.?.objecter ms_handle_connect 0x7ff4d01430c0 2024-03-25T19:42:29.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.550+0000 7ff4c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.551+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886196885 0 0) 0x7ff4d01397c0 con 0x7ff4d013f710 2024-03-25T19:42:29.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.551+0000 7ff4c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.551+0000 7ff4c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.551+0000 7ff4c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.551+0000 7ff4c6ffd640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:42:29.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.551+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b0004eb0 con 0x7ff4d013f710 2024-03-25T19:42:29.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.553+0000 7ff4c6ffd640 10 client.?.objecter ms_handle_connect 0x7ff4d00a98f0 2024-03-25T19:42:29.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.553+0000 7ff4c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307575802 0 0) 0x7ff4d01d7810 con 0x7ff4d01430c0 2024-03-25T19:42:29.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:42:29.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d01397c0 con 0x7ff4d01430c0 2024-03-25T19:42:29.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091546183 0 0) 0x7ff4b0004eb0 con 0x7ff4d013f710 2024-03-25T19:42:29.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4d01d7810 con 0x7ff4d013f710 2024-03-25T19:42:29.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff4b80019d0 con 0x7ff4d013f710 2024-03-25T19:42:29.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:29.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:29.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.565+0000 7ff4c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:29.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.574+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534866765 0 0) 0x7ff4d0138a60 con 0x7ff4d00a98f0 2024-03-25T19:42:29.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.574+0000 7ff4c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.574+0000 7ff4c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.574+0000 7ff4c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.574+0000 7ff4c6ffd640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:42:29.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.574+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b0004eb0 con 0x7ff4d00a98f0 2024-03-25T19:42:29.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.576+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1961076861 0 0) 0x7ff4d01d7810 con 0x7ff4d013f710 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 >> v1:172.21.15.110:6790/0 conn(0x7ff4d01430c0 legacy=0x7ff4d01e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 >> v1:172.21.15.110:6789/0 conn(0x7ff4d00a98f0 legacy=0x7ff4d0138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 1 monclient: found mon.b 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d01d51f0 con 0x7ff4d013f710 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 5 monclient: authenticate success, global_id 4324 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 10 monclient: _renew_subs 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:29.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d01d8440 con 0x7ff4d013f710 2024-03-25T19:42:29.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:29.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 10 monclient: _renew_subs 2024-03-25T19:42:29.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:29.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d0138eb0 con 0x7ff4d013f710 2024-03-25T19:42:29.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.577+0000 7ff4d4ba7640 1 librados: init done 2024-03-25T19:42:29.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.578+0000 7ff4d4ba7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.578+0000 7ff4d4ba7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.578+0000 7ff4d4ba7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:29.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.578+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d01d79a0 con 0x7ff4d013f710 2024-03-25T19:42:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.879+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4b8003610 con 0x7ff4d013f710 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.879+0000 7ff4c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.879+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff4b8003a20 con 0x7ff4d013f710 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.879+0000 7ff4c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.879+0000 7ff4c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.879+0000 7ff4c6ffd640 10 monclient: dump: 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:29.879 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:29.880 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:29.880 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:29.880 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:29.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.879+0000 7ff4cd59a640 4 set_mon_vals no callback set 2024-03-25T19:42:29.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.880+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7ff4b8003c40 con 0x7ff4d013f710 2024-03-25T19:42:29.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.880+0000 7ff4ce59c640 10 monclient: get_auth_request con 0x7ff4b006de00 auth_method 0 2024-03-25T19:42:29.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.880+0000 7ff4c6ffd640 10 client.4324.objecter ms_handle_connect 0x7ff4b006de00 2024-03-25T19:42:29.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.884+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7ff4b8054810 con 0x7ff4d013f710 2024-03-25T19:42:29.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.884+0000 7ff4c6ffd640 10 client.4324.objecter ms_dispatch 0x7ff4d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:29.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.884+0000 7ff4c6ffd640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:29.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.884+0000 7ff4c6ffd640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:29.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.884+0000 7ff4c6ffd640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T19:42:29.885 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.885+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4b8050320 con 0x7ff4d013f710 2024-03-25T19:42:29.885 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.885+0000 7ff4c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.885 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:29.885+0000 7ff4c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:30.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.093+0000 7ff4d4ba7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:30.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.093+0000 7ff4d4ba7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:30.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.093+0000 7ff4d4ba7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:30.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.093+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4d00627b0 con 0x7ff4d013f710 2024-03-25T19:42:30.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.153+0000 7ff4c6ffd640 1 -- 172.21.15.110:0/1098302075 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7ff4b8052250 con 0x7ff4d013f710 2024-03-25T19:42:30.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.153+0000 7ff4c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:30.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.153+0000 7ff4c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:30.153 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:42:30.154 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 10 librados: watch_flush enter 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 10 librados: watch_flush exit 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 10 client.4324.objecter successfully canceled tick 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 >> v1:172.21.15.178:6800/1132739058 conn(0x7ff4b006de00 legacy=0x7ff4b00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 10 monclient: shutdown 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 >> v1:172.21.15.178:6789/0 conn(0x7ff4d013f710 legacy=0x7ff4d01d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 shutdown_connections 2024-03-25T19:42:30.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 >> 172.21.15.110:0/1098302075 conn(0x7ff4d00b7ea0 msgr2=0x7ff4d013d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:30.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 shutdown_connections 2024-03-25T19:42:30.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 1 -- 172.21.15.110:0/1098302075 wait complete. 2024-03-25T19:42:30.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:30.156+0000 7ff4d4ba7640 1 librados: shutdown 2024-03-25T19:42:30.167 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:36.971 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:37.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.085+0000 7f0a37da8640 10 monclient: get_monmap_and_config 2024-03-25T19:42:37.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.085+0000 7f0a37da8640 10 monclient: build_initial_monmap 2024-03-25T19:42:37.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.085+0000 7f0a37da8640 10 monclient: monmap: 2024-03-25T19:42:37.084 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:42:37.086150+0000 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:42:37.086150+0000 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:37.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.085+0000 7f0a37da8640 1 Processor -- start 2024-03-25T19:42:37.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.086+0000 7f0a37da8640 1 -- start start 2024-03-25T19:42:37.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.086+0000 7f0a37da8640 10 monclient: init 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.086+0000 7f0a37da8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.086+0000 7f0a37da8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.086+0000 7f0a37da8640 10 monclient(hunting): picked mon.noname-b con 0x7f0a300b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.086+0000 7f0a37da8640 10 monclient(hunting): picked mon.noname-c con 0x7f0a300b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.086+0000 7f0a37da8640 10 monclient(hunting): picked mon.noname-a con 0x7f0a3014be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.086+0000 7f0a37da8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a300af940 con 0x7f0a3014be90 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.087+0000 7f0a37da8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a300b0a70 con 0x7f0a300b1730 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.087+0000 7f0a37da8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a300b0d50 con 0x7f0a300b4d60 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.087+0000 7f0a37da8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:37.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.087+0000 7f0a37da8640 10 monclient(hunting): authenticate will time out at 4328.227868s 2024-03-25T19:42:37.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.091+0000 7f0a35343640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f0a300b1730 0x7f0a300b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:36238/0 (socket says 172.21.15.110:36238) 2024-03-25T19:42:37.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.091+0000 7f0a35343640 1 -- 172.21.15.110:0/1862013006 learned_addr learned my addr 172.21.15.110:0/1862013006 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:37.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.110+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901658276 0 0) 0x7f0a300b0d50 con 0x7f0a300b4d60 2024-03-25T19:42:37.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.110+0000 7f0a34b42640 10 monclient(hunting): _negotiate 2024-03-25T19:42:37.110 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.110+0000 7f0a34b42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:37.110 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.110+0000 7f0a34b42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:37.110 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.110+0000 7f0a34b42640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:42:37.110 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.110+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a1c003520 con 0x7f0a300b4d60 2024-03-25T19:42:37.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.112+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206772444 0 0) 0x7f0a300b0a70 con 0x7f0a300b1730 2024-03-25T19:42:37.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.112+0000 7f0a34b42640 10 monclient(hunting): _negotiate 2024-03-25T19:42:37.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.112+0000 7f0a34b42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:37.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.112+0000 7f0a34b42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:37.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.112+0000 7f0a34b42640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:42:37.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.112+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a300b0d50 con 0x7f0a300b1730 2024-03-25T19:42:37.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 560646444 0 0) 0x7f0a300b0d50 con 0x7f0a300b1730 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 >> v1:172.21.15.110:6790/0 conn(0x7f0a300b4d60 legacy=0x7f0a3013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 >> v1:172.21.15.110:6789/0 conn(0x7f0a3014be90 legacy=0x7f0a3014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 10 monclient: _finish_hunting 0 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 1 monclient: found mon.noname-c 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a300b1030 con 0x7f0a300b1730 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 10 monclient: _finish_auth 0 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 10 monclient: _check_auth_tickets 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0a20001670 con 0x7f0a300b1730 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:37.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a34b42640 10 monclient: dump: 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a37da8640 5 monclient: authenticate success, global_id 4327 2024-03-25T19:42:37.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.123+0000 7f0a37da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:37.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.133+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a200017d0 con 0x7f0a300b1730 2024-03-25T19:42:37.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.133+0000 7f0a34b42640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:37.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.133+0000 7f0a34b42640 1 -- 172.21.15.110:0/1862013006 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0a20001b60 con 0x7f0a300b1730 2024-03-25T19:42:37.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.133+0000 7f0a37da8640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:37.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.133+0000 7f0a37da8640 4 set_mon_vals no callback set 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.133+0000 7f0a34b42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.133+0000 7f0a34b42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.133+0000 7f0a34b42640 10 monclient: dump: 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:37.133 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.134+0000 7f0a37da8640 10 monclient: shutdown 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.134+0000 7f0a37da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.134+0000 7f0a37da8640 1 -- 172.21.15.110:0/1862013006 >> v1:172.21.15.178:6789/0 conn(0x7f0a300b1730 legacy=0x7f0a300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.134+0000 7f0a37da8640 1 -- 172.21.15.110:0/1862013006 shutdown_connections 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.134+0000 7f0a37da8640 1 -- 172.21.15.110:0/1862013006 >> 172.21.15.110:0/1862013006 conn(0x7f0a300617a0 msgr2=0x7f0a30061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.134+0000 7f0a37da8640 1 -- 172.21.15.110:0/1862013006 shutdown_connections 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.134+0000 7f0a37da8640 1 -- 172.21.15.110:0/1862013006 wait complete. 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 10 monclient: build_initial_monmap 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 10 monclient: monmap: 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:42:37.136 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 1 librados: starting msgr at 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 1 librados: starting objecter 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 1 Processor -- start 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 1 -- start start 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 1 librados: setting wanted keys 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 1 librados: calling monclient init 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.135+0000 7f0a37da8640 10 monclient: init 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 10 monclient(hunting): picked mon.noname-b con 0x7f0a300b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:37.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 10 monclient(hunting): picked mon.noname-a con 0x7f0a300b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:37.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 10 monclient(hunting): picked mon.noname-c con 0x7f0a3014be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:37.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a301307c0 con 0x7f0a300b4d60 2024-03-25T19:42:37.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a30145380 con 0x7f0a300b1730 2024-03-25T19:42:37.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a30145010 con 0x7f0a3014be90 2024-03-25T19:42:37.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:37.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.136+0000 7f0a37da8640 10 monclient(hunting): authenticate will time out at 4328.277268s 2024-03-25T19:42:37.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.142+0000 7f0a36345640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f0a3014be90 0x7f0a301300b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:57078/0 (socket says 172.21.15.110:57078) 2024-03-25T19:42:37.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.142+0000 7f0a36345640 1 -- 172.21.15.110:0/3857611658 learned_addr learned my addr 172.21.15.110:0/3857611658 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:37.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.144+0000 7f0a267fc640 10 client.?.objecter ms_handle_connect 0x7f0a3014be90 2024-03-25T19:42:37.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.144+0000 7f0a267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:37.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.145+0000 7f0a267fc640 10 client.?.objecter ms_handle_connect 0x7f0a300b1730 2024-03-25T19:42:37.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.145+0000 7f0a267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:37.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.153+0000 7f0a267fc640 10 client.?.objecter ms_handle_connect 0x7f0a300b4d60 2024-03-25T19:42:37.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.153+0000 7f0a267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:37.200 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:44.484 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.210 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:44.494 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.216+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334820066 0 0) 0x7f0a301307c0 con 0x7f0a300b4d60 2024-03-25T19:42:37.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.216+0000 7f0a267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:37.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.216+0000 7f0a267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:37.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.216+0000 7f0a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:37.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.216+0000 7f0a267fc640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:42:37.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.216+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a14004eb0 con 0x7f0a300b4d60 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.236+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 36098824 0 0) 0x7f0a14004eb0 con 0x7f0a300b4d60 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.236+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a301307c0 con 0x7f0a300b4d60 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.236+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0a20001ae0 con 0x7f0a300b4d60 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.236+0000 7f0a267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.236+0000 7f0a267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.236+0000 7f0a267fc640 10 monclient(hunting): dump: 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:37.236 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.237 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:37.237 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:37.237 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:37.237 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:37.237 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:37.237 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.236+0000 7f0a267fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:37.246 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:44.530 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.271 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:44.526 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.272+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346061020 0 0) 0x7f0a30145380 con 0x7f0a300b1730 2024-03-25T19:42:37.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.272+0000 7f0a267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:37.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.272+0000 7f0a267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:37.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.272+0000 7f0a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:37.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.272+0000 7f0a267fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:42:37.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.272+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a14004eb0 con 0x7f0a300b1730 2024-03-25T19:42:37.274 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:44.529 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.275 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:44.559 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.276+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191815675 0 0) 0x7f0a30145010 con 0x7f0a3014be90 2024-03-25T19:42:37.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.276+0000 7f0a267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:37.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.276+0000 7f0a267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:37.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.276+0000 7f0a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:37.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.276+0000 7f0a267fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:42:37.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.276+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a30145380 con 0x7f0a3014be90 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656770659 0 0) 0x7f0a301307c0 con 0x7f0a300b4d60 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 >> v1:172.21.15.110:6790/0 conn(0x7f0a3014be90 legacy=0x7f0a301300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 >> v1:172.21.15.178:6789/0 conn(0x7f0a300b1730 legacy=0x7f0a301dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 1 monclient: found mon.a 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a30137ad0 con 0x7f0a300b4d60 2024-03-25T19:42:37.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 10 monclient: _renew_subs 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a301360e0 con 0x7f0a300b4d60 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 10 monclient: _renew_subs 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a3013e2d0 con 0x7f0a300b4d60 2024-03-25T19:42:37.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.279+0000 7f0a37da8640 1 librados: init done 2024-03-25T19:42:37.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.280+0000 7f0a37da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:37.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.280+0000 7f0a37da8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:37.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.280+0000 7f0a37da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:37.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.280+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a30137d20 con 0x7f0a300b4d60 2024-03-25T19:42:37.284 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:44.568 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.302 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:02:44.556 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.307 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:44.591 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.577+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a20004a90 con 0x7f0a300b4d60 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.577+0000 7f0a267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.577+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0a20004da0 con 0x7f0a300b4d60 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.577+0000 7f0a267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.577+0000 7f0a267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.577+0000 7f0a267fc640 10 monclient: dump: 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:37.577 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:37.578 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:37.578 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:37.578 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:37.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.577+0000 7f0a27fff640 4 set_mon_vals no callback set 2024-03-25T19:42:37.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.578+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f0a2001ad70 con 0x7f0a300b4d60 2024-03-25T19:42:37.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.579+0000 7f0a35b44640 10 monclient: get_auth_request con 0x7f0a1406ddf0 auth_method 0 2024-03-25T19:42:37.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.579+0000 7f0a267fc640 10 client.4320.objecter ms_handle_connect 0x7f0a1406ddf0 2024-03-25T19:42:37.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.584+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f0a200548e0 con 0x7f0a300b4d60 2024-03-25T19:42:37.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.584+0000 7f0a267fc640 10 client.4320.objecter ms_dispatch 0x7f0a30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:37.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.584+0000 7f0a267fc640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:37.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.584+0000 7f0a267fc640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:37.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.584+0000 7f0a267fc640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-25T19:42:37.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.584+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a20050220 con 0x7f0a300b4d60 2024-03-25T19:42:37.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.584+0000 7f0a267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:37.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.584+0000 7f0a267fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:37.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.833+0000 7f0a37da8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.833+0000 7f0a37da8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.833+0000 7f0a37da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:37.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.833+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a30145510 con 0x7f0a300b4d60 2024-03-25T19:42:37.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.869+0000 7f0a267fc640 1 -- 172.21.15.110:0/3857611658 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f0a20052320 con 0x7f0a300b4d60 2024-03-25T19:42:37.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.869+0000 7f0a267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.869+0000 7f0a267fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:37.869 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:42:37.869 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:37.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 10 librados: watch_flush enter 2024-03-25T19:42:37.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 10 librados: watch_flush exit 2024-03-25T19:42:37.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-25T19:42:37.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 10 client.4320.objecter successfully canceled tick 2024-03-25T19:42:37.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 >> v1:172.21.15.178:6800/1132739058 conn(0x7f0a1406ddf0 legacy=0x7f0a140702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 10 monclient: shutdown 2024-03-25T19:42:37.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:37.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 >> v1:172.21.15.110:6789/0 conn(0x7f0a300b4d60 legacy=0x7f0a3012f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 shutdown_connections 2024-03-25T19:42:37.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 >> 172.21.15.110:0/3857611658 conn(0x7f0a300617a0 msgr2=0x7f0a300b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:37.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 shutdown_connections 2024-03-25T19:42:37.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 1 -- 172.21.15.110:0/3857611658 wait complete. 2024-03-25T19:42:37.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:37.872+0000 7f0a37da8640 1 librados: shutdown 2024-03-25T19:42:37.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:44.691 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.809+0000 7f42e80ff640 10 monclient: get_monmap_and_config 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.809+0000 7f42e80ff640 10 monclient: build_initial_monmap 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.809+0000 7f42e80ff640 10 monclient: monmap: 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:42:44.811022+0000 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:42:44.811022+0000 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:44.810 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:42:44.811 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:42:44.811 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:44.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.809+0000 7f42e80ff640 1 Processor -- start 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 1 -- start start 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 10 monclient: init 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 10 monclient(hunting): picked mon.noname-a con 0x7f42e00a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 10 monclient(hunting): picked mon.noname-c con 0x7f42e013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 10 monclient(hunting): picked mon.noname-b con 0x7f42e01430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e0142520 con 0x7f42e00a98f0 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00b28a0 con 0x7f42e013f710 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00b2a30 con 0x7f42e01430c0 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:44.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.810+0000 7f42e80ff640 10 monclient(hunting): authenticate will time out at 4335.952626s 2024-03-25T19:42:44.815 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.814+0000 7f42e569a640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f42e013f710 0x7f42e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:42350/0 (socket says 172.21.15.110:42350) 2024-03-25T19:42:44.815 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.814+0000 7f42e569a640 1 -- 172.21.15.110:0/2192318264 learned_addr learned my addr 172.21.15.110:0/2192318264 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:44.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.821+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668195673 0 0) 0x7f42e00b28a0 con 0x7f42e013f710 2024-03-25T19:42:44.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.821+0000 7f42e4e99640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.821+0000 7f42e4e99640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.821+0000 7f42e4e99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.821+0000 7f42e4e99640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:42:44.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.821+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42cc003520 con 0x7f42e013f710 2024-03-25T19:42:44.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.843+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126024966 0 0) 0x7f42e0142520 con 0x7f42e00a98f0 2024-03-25T19:42:44.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.843+0000 7f42e4e99640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.843+0000 7f42e4e99640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.843+0000 7f42e4e99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.843+0000 7f42e4e99640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:42:44.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.843+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e00b28a0 con 0x7f42e00a98f0 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 476618770 0 0) 0x7f42e00b28a0 con 0x7f42e00a98f0 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 >> v1:172.21.15.110:6790/0 conn(0x7f42e01430c0 legacy=0x7f42e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 >> v1:172.21.15.178:6789/0 conn(0x7f42e013f710 legacy=0x7f42e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 10 monclient: _finish_hunting 0 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 1 monclient: found mon.noname-a 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e00b44c0 con 0x7f42e00a98f0 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 10 monclient: _finish_auth 0 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 10 monclient: _check_auth_tickets 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f42dc001670 con 0x7f42e00a98f0 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e80ff640 5 monclient: authenticate success, global_id 4323 2024-03-25T19:42:44.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e80ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e4e99640 10 monclient: dump: 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:44.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.913+0000 7f42e80ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.933+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42dc0017d0 con 0x7f42e00a98f0 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.933+0000 7f42e4e99640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.933+0000 7f42e4e99640 1 -- 172.21.15.110:0/2192318264 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f42dc001ba0 con 0x7f42e00a98f0 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.933+0000 7f42e80ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.933+0000 7f42e80ff640 4 set_mon_vals no callback set 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.933+0000 7f42e4e99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.933+0000 7f42e4e99640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.933+0000 7f42e4e99640 10 monclient: dump: 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:44.934 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:44.935 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:44.935 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:44.935 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.935 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:44.935 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:44.935 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:44.935 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:44.935 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.934+0000 7f42e80ff640 10 monclient: shutdown 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.934+0000 7f42e80ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.934+0000 7f42e80ff640 1 -- 172.21.15.110:0/2192318264 >> v1:172.21.15.110:6789/0 conn(0x7f42e00a98f0 legacy=0x7f42e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.934+0000 7f42e80ff640 1 -- 172.21.15.110:0/2192318264 shutdown_connections 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.934+0000 7f42e80ff640 1 -- 172.21.15.110:0/2192318264 >> 172.21.15.110:0/2192318264 conn(0x7f42e00b7ea0 msgr2=0x7f42e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.934+0000 7f42e80ff640 1 -- 172.21.15.110:0/2192318264 shutdown_connections 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.934+0000 7f42e80ff640 1 -- 172.21.15.110:0/2192318264 wait complete. 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.935+0000 7f42e80ff640 10 monclient: build_initial_monmap 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.935+0000 7f42e80ff640 10 monclient: monmap: 2024-03-25T19:42:44.936 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:42:44.937 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:44.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.936+0000 7f42e80ff640 1 librados: starting msgr at 2024-03-25T19:42:44.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.936+0000 7f42e80ff640 1 librados: starting objecter 2024-03-25T19:42:44.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.936+0000 7f42e80ff640 1 Processor -- start 2024-03-25T19:42:44.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.936+0000 7f42e80ff640 1 -- start start 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.936+0000 7f42e80ff640 1 librados: setting wanted keys 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.936+0000 7f42e80ff640 1 librados: calling monclient init 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.936+0000 7f42e80ff640 10 monclient: init 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 10 monclient(hunting): picked mon.noname-c con 0x7f42e00a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 10 monclient(hunting): picked mon.noname-b con 0x7f42e013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 10 monclient(hunting): picked mon.noname-a con 0x7f42e01430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e013bf90 con 0x7f42e01430c0 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e013b060 con 0x7f42e013f710 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e01dc070 con 0x7f42e00a98f0 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:44.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.937+0000 7f42e80ff640 10 monclient(hunting): authenticate will time out at 4336.079015s 2024-03-25T19:42:44.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.940+0000 7f42e569a640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f42e013f710 0x7f42e01db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:42390/0 (socket says 172.21.15.110:42390) 2024-03-25T19:42:44.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.940+0000 7f42e569a640 1 -- 172.21.15.110:0/4058091443 learned_addr learned my addr 172.21.15.110:0/4058091443 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:44.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.941+0000 7f42d67fc640 10 client.?.objecter ms_handle_connect 0x7f42e013f710 2024-03-25T19:42:44.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.941+0000 7f42d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.944+0000 7f42d67fc640 10 client.?.objecter ms_handle_connect 0x7f42e01430c0 2024-03-25T19:42:44.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.944+0000 7f42d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.948+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636475660 0 0) 0x7f42e013b060 con 0x7f42e013f710 2024-03-25T19:42:44.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.949+0000 7f42d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.949+0000 7f42d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.949+0000 7f42d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.949+0000 7f42d67fc640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:42:44.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.949+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c4004eb0 con 0x7f42e013f710 2024-03-25T19:42:44.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.957+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969957617 0 0) 0x7f42e013bf90 con 0x7f42e01430c0 2024-03-25T19:42:44.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.957+0000 7f42d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.957+0000 7f42d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.957+0000 7f42d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.957+0000 7f42d67fc640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:42:44.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.957+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e013b060 con 0x7f42e01430c0 2024-03-25T19:42:44.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.958+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3095421716 0 0) 0x7f42c4004eb0 con 0x7f42e013f710 2024-03-25T19:42:44.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.958+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42e013bf90 con 0x7f42e013f710 2024-03-25T19:42:44.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.958+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f42d0001950 con 0x7f42e013f710 2024-03-25T19:42:44.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.958+0000 7f42d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.958+0000 7f42d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.958+0000 7f42d67fc640 10 monclient(hunting): dump: 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:44.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.958+0000 7f42d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:44.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1096396508 0 0) 0x7f42e013bf90 con 0x7f42e013f710 2024-03-25T19:42:44.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 >> v1:172.21.15.110:6790/0 conn(0x7f42e00a98f0 legacy=0x7f42e013c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 >> v1:172.21.15.110:6789/0 conn(0x7f42e01430c0 legacy=0x7f42e01db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:44.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 1 monclient: found mon.b 2024-03-25T19:42:44.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:44.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:44.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e013b2d0 con 0x7f42e013f710 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 10 monclient: _renew_subs 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e013b5b0 con 0x7f42e013f710 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 10 monclient: _renew_subs 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e00b7660 con 0x7f42e013f710 2024-03-25T19:42:44.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.971+0000 7f42e80ff640 1 librados: init done 2024-03-25T19:42:44.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.973+0000 7f42e80ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.973+0000 7f42e80ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.973+0000 7f42e80ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:44.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:44.973+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e00b3ef0 con 0x7f42e013f710 2024-03-25T19:42:45.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.263+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42d00047e0 con 0x7f42e013f710 2024-03-25T19:42:45.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.263+0000 7f42d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:45.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.263+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f42d0001d80 con 0x7f42e013f710 2024-03-25T19:42:45.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.263+0000 7f42d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:45.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.263+0000 7f42d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.263+0000 7f42d67fc640 10 monclient: dump: 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:45.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.263+0000 7f42d7fff640 4 set_mon_vals no callback set 2024-03-25T19:42:45.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.265+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f42d0004c30 con 0x7f42e013f710 2024-03-25T19:42:45.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.265+0000 7f42e5e9b640 10 monclient: get_auth_request con 0x7f42c406dcb0 auth_method 0 2024-03-25T19:42:45.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.265+0000 7f42d67fc640 10 client.4336.objecter ms_handle_connect 0x7f42c406dcb0 2024-03-25T19:42:45.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.268+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f42d00547c0 con 0x7f42e013f710 2024-03-25T19:42:45.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.268+0000 7f42d67fc640 10 client.4336.objecter ms_dispatch 0x7f42e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:45.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.268+0000 7f42d67fc640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:45.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.268+0000 7f42d67fc640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:45.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.268+0000 7f42d67fc640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T19:42:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.270+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42d00502d0 con 0x7f42e013f710 2024-03-25T19:42:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.270+0000 7f42d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.270+0000 7f42d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:45.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.520+0000 7f42e80ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.520+0000 7f42e80ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.520+0000 7f42e80ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:42:45.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.520+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42e00627b0 con 0x7f42e013f710 2024-03-25T19:42:45.555 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.554+0000 7f42d67fc640 1 -- 172.21.15.110:0/4058091443 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f42d0052200 con 0x7f42e013f710 2024-03-25T19:42:45.555 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.554+0000 7f42d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.555 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.554+0000 7f42d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:45.555 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:42:45.555 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 10 librados: watch_flush enter 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 10 librados: watch_flush exit 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 10 client.4336.objecter successfully canceled tick 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 >> v1:172.21.15.178:6800/1132739058 conn(0x7f42c406dcb0 legacy=0x7f42c4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 10 monclient: shutdown 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 >> v1:172.21.15.178:6789/0 conn(0x7f42e013f710 legacy=0x7f42e01db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 shutdown_connections 2024-03-25T19:42:45.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 >> 172.21.15.110:0/4058091443 conn(0x7f42e00b7ea0 msgr2=0x7f42e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:45.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 shutdown_connections 2024-03-25T19:42:45.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 1 -- 172.21.15.110:0/4058091443 wait complete. 2024-03-25T19:42:45.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:45.557+0000 7f42e80ff640 1 librados: shutdown 2024-03-25T19:42:45.570 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:52.377 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:52.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.494+0000 7efe2a293640 10 monclient: get_monmap_and_config 2024-03-25T19:42:52.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.494+0000 7efe2a293640 10 monclient: build_initial_monmap 2024-03-25T19:42:52.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.494+0000 7efe2a293640 10 monclient: monmap: 2024-03-25T19:42:52.494 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:52.494 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:52.494 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:42:52.495646+0000 2024-03-25T19:42:52.494 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:42:52.495646+0000 2024-03-25T19:42:52.495 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:52.495 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:52.495 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:52.495 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:42:52.495 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:42:52.495 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.495+0000 7efe2a293640 1 Processor -- start 2024-03-25T19:42:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.495+0000 7efe2a293640 1 -- start start 2024-03-25T19:42:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.495+0000 7efe2a293640 10 monclient: init 2024-03-25T19:42:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 10 monclient(hunting): picked mon.noname-c con 0x7efe240a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 10 monclient(hunting): picked mon.noname-a con 0x7efe2413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 10 monclient(hunting): picked mon.noname-b con 0x7efe241430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe24142520 con 0x7efe2413f710 2024-03-25T19:42:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe240b28a0 con 0x7efe240a98f0 2024-03-25T19:42:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe240b2a30 con 0x7efe241430c0 2024-03-25T19:42:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.496+0000 7efe2a293640 10 monclient(hunting): authenticate will time out at 4343.637668s 2024-03-25T19:42:52.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.508+0000 7efe28830640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7efe241430c0 0x7efe2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:34684/0 (socket says 172.21.15.110:34684) 2024-03-25T19:42:52.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.508+0000 7efe28830640 1 -- 172.21.15.110:0/3376018535 learned_addr learned my addr 172.21.15.110:0/3376018535 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:52.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.518+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779566586 0 0) 0x7efe240b2a30 con 0x7efe241430c0 2024-03-25T19:42:52.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.518+0000 7efe22ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.518+0000 7efe22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.518+0000 7efe22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.518+0000 7efe22ffd640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:42:52.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.518+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c003520 con 0x7efe241430c0 2024-03-25T19:42:52.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.528+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148961282 0 0) 0x7efe24142520 con 0x7efe2413f710 2024-03-25T19:42:52.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.528+0000 7efe22ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.528+0000 7efe22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.528+0000 7efe22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.528+0000 7efe22ffd640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:42:52.528 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.528+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe240b2a30 con 0x7efe2413f710 2024-03-25T19:42:52.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.533+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073166701 0 0) 0x7efe240b28a0 con 0x7efe240a98f0 2024-03-25T19:42:52.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.533+0000 7efe22ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.533+0000 7efe22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.533+0000 7efe22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.533+0000 7efe22ffd640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:42:52.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.533+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe24142520 con 0x7efe240a98f0 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 31299799 0 0) 0x7efe24142520 con 0x7efe240a98f0 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 >> v1:172.21.15.110:6790/0 conn(0x7efe241430c0 legacy=0x7efe2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 >> v1:172.21.15.110:6789/0 conn(0x7efe2413f710 legacy=0x7efe241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 1 monclient: found mon.noname-c 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240b44c0 con 0x7efe240a98f0 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efe140015b0 con 0x7efe240a98f0 2024-03-25T19:42:52.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe2a293640 5 monclient: authenticate success, global_id 4339 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe2a293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe22ffd640 10 monclient: dump: 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:52.546 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.545+0000 7efe2a293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.562+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe14001710 con 0x7efe240a98f0 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.562+0000 7efe22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.562+0000 7efe22ffd640 1 -- 172.21.15.110:0/3376018535 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efe14001b60 con 0x7efe240a98f0 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.562+0000 7efe22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.562+0000 7efe22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.562+0000 7efe22ffd640 10 monclient: dump: 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:52.562 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:52.563 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.563 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:52.563 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:52.563 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:52.563 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:52.563 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:52.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.562+0000 7efe2a293640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:52.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.562+0000 7efe2a293640 4 set_mon_vals no callback set 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.563+0000 7efe2a293640 10 monclient: shutdown 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.563+0000 7efe2a293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.563+0000 7efe2a293640 1 -- 172.21.15.110:0/3376018535 >> v1:172.21.15.178:6789/0 conn(0x7efe240a98f0 legacy=0x7efe240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.563+0000 7efe2a293640 1 -- 172.21.15.110:0/3376018535 shutdown_connections 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.563+0000 7efe2a293640 1 -- 172.21.15.110:0/3376018535 >> 172.21.15.110:0/3376018535 conn(0x7efe240b7ea0 msgr2=0x7efe240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.563+0000 7efe2a293640 1 -- 172.21.15.110:0/3376018535 shutdown_connections 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.563+0000 7efe2a293640 1 -- 172.21.15.110:0/3376018535 wait complete. 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.564+0000 7efe2a293640 10 monclient: build_initial_monmap 2024-03-25T19:42:52.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.564+0000 7efe2a293640 10 monclient: monmap: 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.564+0000 7efe2a293640 1 librados: starting msgr at 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.564+0000 7efe2a293640 1 librados: starting objecter 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.564+0000 7efe2a293640 1 Processor -- start 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 1 -- start start 2024-03-25T19:42:52.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 1 librados: setting wanted keys 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 1 librados: calling monclient init 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 10 monclient: init 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 10 monclient(hunting): picked mon.noname-a con 0x7efe240a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 10 monclient(hunting): picked mon.noname-b con 0x7efe2413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 10 monclient(hunting): picked mon.noname-c con 0x7efe241430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe241df240 con 0x7efe240a98f0 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe240706d0 con 0x7efe2413f710 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe241dfde0 con 0x7efe241430c0 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:52.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.565+0000 7efe2a293640 10 monclient(hunting): authenticate will time out at 4343.706710s 2024-03-25T19:42:52.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.566+0000 7efe23fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7efe240a98f0 0x7efe241df6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33150/0 (socket says 172.21.15.110:33150) 2024-03-25T19:42:52.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.566+0000 7efe23fff640 1 -- 172.21.15.110:0/1258085701 learned_addr learned my addr 172.21.15.110:0/1258085701 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:42:52.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.569+0000 7efe20ff9640 10 client.?.objecter ms_handle_connect 0x7efe240a98f0 2024-03-25T19:42:52.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.569+0000 7efe20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.582+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468768770 0 0) 0x7efe241df240 con 0x7efe240a98f0 2024-03-25T19:42:52.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.582+0000 7efe20ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.582+0000 7efe20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.582+0000 7efe20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.582+0000 7efe20ff9640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:42:52.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.582+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe10004eb0 con 0x7efe240a98f0 2024-03-25T19:42:52.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.586+0000 7efe20ff9640 10 client.?.objecter ms_handle_connect 0x7efe2413f710 2024-03-25T19:42:52.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.586+0000 7efe20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.594 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.594+0000 7efe20ff9640 10 client.?.objecter ms_handle_connect 0x7efe241430c0 2024-03-25T19:42:52.594 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.594+0000 7efe20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.601+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3542382881 0 0) 0x7efe10004eb0 con 0x7efe240a98f0 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.601+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe241df240 con 0x7efe240a98f0 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.601+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efe14001b20 con 0x7efe240a98f0 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.601+0000 7efe20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.601+0000 7efe20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.601+0000 7efe20ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:52.601 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:52.602 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.602 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:52.602 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:52.602 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:52.602 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:52.602 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:52.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.601+0000 7efe20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:52.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.604+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620375001 0 0) 0x7efe240706d0 con 0x7efe2413f710 2024-03-25T19:42:52.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.604+0000 7efe20ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.604+0000 7efe20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.604+0000 7efe20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.604+0000 7efe20ff9640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:42:52.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.604+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe10004eb0 con 0x7efe2413f710 2024-03-25T19:42:52.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1621559549 0 0) 0x7efe241df240 con 0x7efe240a98f0 2024-03-25T19:42:52.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 >> v1:172.21.15.110:6790/0 conn(0x7efe241430c0 legacy=0x7efe2406f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 >> v1:172.21.15.178:6789/0 conn(0x7efe2413f710 legacy=0x7efe2406eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 1 monclient: found mon.a 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe241dff70 con 0x7efe240a98f0 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.605+0000 7efe20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 10 monclient: _renew_subs 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe24070dd0 con 0x7efe240a98f0 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 10 monclient: _renew_subs 2024-03-25T19:42:52.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:52.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe241e7910 con 0x7efe240a98f0 2024-03-25T19:42:52.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.606+0000 7efe2a293640 1 librados: init done 2024-03-25T19:42:52.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.607+0000 7efe2a293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.607+0000 7efe2a293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.607+0000 7efe2a293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:52.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.607+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe240627b0 con 0x7efe240a98f0 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.904+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe140038a0 con 0x7efe240a98f0 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.904+0000 7efe20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.904+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efe14003cb0 con 0x7efe240a98f0 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.904+0000 7efe20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.904+0000 7efe20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.904+0000 7efe20ff9640 10 monclient: dump: 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:42:52.904 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:52.905 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:42:52.905 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.905 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:42:52.905 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:42:52.905 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:42:52.905 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:42:52.905 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:42:52.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.904+0000 7efe227fc640 4 set_mon_vals no callback set 2024-03-25T19:42:52.906 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.905+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7efe14003ed0 con 0x7efe240a98f0 2024-03-25T19:42:52.906 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.905+0000 7efe237fe640 10 monclient: get_auth_request con 0x7efe1006dd90 auth_method 0 2024-03-25T19:42:52.906 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.905+0000 7efe20ff9640 10 client.4332.objecter ms_handle_connect 0x7efe1006dd90 2024-03-25T19:42:52.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.912+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7efe14054900 con 0x7efe240a98f0 2024-03-25T19:42:52.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.912+0000 7efe20ff9640 10 client.4332.objecter ms_dispatch 0x7efe24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:52.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.913+0000 7efe20ff9640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:52.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.913+0000 7efe20ff9640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:52.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.913+0000 7efe20ff9640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T19:42:52.912 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.913+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe14050410 con 0x7efe240a98f0 2024-03-25T19:42:52.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.913+0000 7efe20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:52.913+0000 7efe20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:53.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.124+0000 7efe2a293640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:53.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.124+0000 7efe2a293640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:53.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.124+0000 7efe2a293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:42:53.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.124+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe2413de50 con 0x7efe240a98f0 2024-03-25T19:42:53.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.156+0000 7efe20ff9640 1 -- 172.21.15.110:0/1258085701 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7efe14052340 con 0x7efe240a98f0 2024-03-25T19:42:53.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.156+0000 7efe20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:53.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.156+0000 7efe20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:53.156 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:42:53.156 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:53.158 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 10 librados: watch_flush enter 2024-03-25T19:42:53.158 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 10 librados: watch_flush exit 2024-03-25T19:42:53.158 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 10 client.4332.objecter successfully canceled tick 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 >> v1:172.21.15.178:6800/1132739058 conn(0x7efe1006dd90 legacy=0x7efe10070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 10 monclient: shutdown 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 >> v1:172.21.15.110:6789/0 conn(0x7efe240a98f0 legacy=0x7efe241df6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 shutdown_connections 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 >> 172.21.15.110:0/1258085701 conn(0x7efe240b7ea0 msgr2=0x7efe24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 shutdown_connections 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.158+0000 7efe2a293640 1 -- 172.21.15.110:0/1258085701 wait complete. 2024-03-25T19:42:53.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:42:53.159+0000 7efe2a293640 1 librados: shutdown 2024-03-25T19:42:53.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:57.244 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:04.528 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.250 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:04.534 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.284 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:04.568 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.292 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:04.546 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.296 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:04.551 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.332 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:04.587 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.358 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:04.642 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.364 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:04.648 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.401 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:04.685 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.977 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:00.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.091+0000 7f4200315640 10 monclient: get_monmap_and_config 2024-03-25T19:43:00.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.091+0000 7f4200315640 10 monclient: build_initial_monmap 2024-03-25T19:43:00.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.091+0000 7f4200315640 10 monclient: monmap: 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:43:00.092841+0000 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:43:00.092841+0000 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:43:00.092 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.092+0000 7f4200315640 1 Processor -- start 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.092+0000 7f4200315640 1 -- start start 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.092+0000 7f4200315640 10 monclient: init 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 10 monclient(hunting): picked mon.noname-c con 0x7f41f80a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 10 monclient(hunting): picked mon.noname-a con 0x7f41f813f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 10 monclient(hunting): picked mon.noname-b con 0x7f41f81430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f8142520 con 0x7f41f813f710 2024-03-25T19:43:00.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f80b28a0 con 0x7f41f80a98f0 2024-03-25T19:43:00.094 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f80b2a30 con 0x7f41f81430c0 2024-03-25T19:43:00.094 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:00.094 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.093+0000 7f4200315640 10 monclient(hunting): authenticate will time out at 4351.234646s 2024-03-25T19:43:00.110 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.110+0000 7f41fe8b2640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f41f81430c0 0x7f41f814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:39558/0 (socket says 172.21.15.110:39558) 2024-03-25T19:43:00.110 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.110+0000 7f41fe8b2640 1 -- 172.21.15.110:0/1544220969 learned_addr learned my addr 172.21.15.110:0/1544220969 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:00.128 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.129+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97124043 0 0) 0x7f41f80b2a30 con 0x7f41f81430c0 2024-03-25T19:43:00.128 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.129+0000 7f41fd0af640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.128 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.129+0000 7f41fd0af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.129+0000 7f41fd0af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.129+0000 7f41fd0af640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:43:00.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.129+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e4003520 con 0x7f41f81430c0 2024-03-25T19:43:00.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.138+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194813426 0 0) 0x7f41f80b28a0 con 0x7f41f80a98f0 2024-03-25T19:43:00.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.138+0000 7f41fd0af640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.138+0000 7f41fd0af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.138+0000 7f41fd0af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.138+0000 7f41fd0af640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:43:00.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.138+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f80b2a30 con 0x7f41f80a98f0 2024-03-25T19:43:00.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.143+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48306011 0 0) 0x7f41f8142520 con 0x7f41f813f710 2024-03-25T19:43:00.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.143+0000 7f41fd0af640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.143+0000 7f41fd0af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.143+0000 7f41fd0af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.143+0000 7f41fd0af640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:43:00.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.143+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f80b28a0 con 0x7f41f813f710 2024-03-25T19:43:00.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340766204 0 0) 0x7f41e4003520 con 0x7f41f81430c0 2024-03-25T19:43:00.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 >> v1:172.21.15.178:6789/0 conn(0x7f41f80a98f0 legacy=0x7f41f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 >> v1:172.21.15.110:6789/0 conn(0x7f41f813f710 legacy=0x7f41f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 10 monclient: _finish_hunting 0 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 1 monclient: found mon.noname-b 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f80b44c0 con 0x7f41f81430c0 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 10 monclient: _finish_auth 0 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 10 monclient: _check_auth_tickets 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f41f0001a10 con 0x7f41f81430c0 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f41fd0af640 10 monclient: dump: 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f4200315640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.153+0000 7f4200315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:00.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.170+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41f0001b70 con 0x7f41f81430c0 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.170+0000 7f41fd0af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.170+0000 7f41fd0af640 1 -- 172.21.15.110:0/1544220969 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f41f0001f80 con 0x7f41f81430c0 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.170+0000 7f41fd0af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.170+0000 7f41fd0af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.170+0000 7f41fd0af640 10 monclient: dump: 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:00.170 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:00.171 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:00.171 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:00.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.170+0000 7f4200315640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:00.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.170+0000 7f4200315640 4 set_mon_vals no callback set 2024-03-25T19:43:00.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.171+0000 7f4200315640 10 monclient: shutdown 2024-03-25T19:43:00.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.171+0000 7f4200315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:00.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.171+0000 7f4200315640 1 -- 172.21.15.110:0/1544220969 >> v1:172.21.15.110:6790/0 conn(0x7f41f81430c0 legacy=0x7f41f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.171+0000 7f4200315640 1 -- 172.21.15.110:0/1544220969 shutdown_connections 2024-03-25T19:43:00.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.171+0000 7f4200315640 1 -- 172.21.15.110:0/1544220969 >> 172.21.15.110:0/1544220969 conn(0x7f41f80b7ea0 msgr2=0x7f41f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:00.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.171+0000 7f4200315640 1 -- 172.21.15.110:0/1544220969 shutdown_connections 2024-03-25T19:43:00.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.171+0000 7f4200315640 1 -- 172.21.15.110:0/1544220969 wait complete. 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 10 monclient: build_initial_monmap 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 10 monclient: monmap: 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 1 librados: starting msgr at 2024-03-25T19:43:00.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 1 librados: starting objecter 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 1 Processor -- start 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 1 -- start start 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 1 librados: setting wanted keys 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 1 librados: calling monclient init 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.172+0000 7f4200315640 10 monclient: init 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 10 monclient(hunting): picked mon.noname-b con 0x7f41f80a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 10 monclient(hunting): picked mon.noname-c con 0x7f41f813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 10 monclient(hunting): picked mon.noname-a con 0x7f41f81430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f813c9b0 con 0x7f41f81430c0 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f81397c0 con 0x7f41f80a98f0 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f8139950 con 0x7f41f813f710 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:00.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.173+0000 7f4200315640 10 monclient(hunting): authenticate will time out at 4351.314251s 2024-03-25T19:43:00.179 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.179+0000 7f41fe8b2640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f41f81430c0 0x7f41f81e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37812/0 (socket says 172.21.15.110:37812) 2024-03-25T19:43:00.179 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.179+0000 7f41fe8b2640 1 -- 172.21.15.110:0/1143178920 learned_addr learned my addr 172.21.15.110:0/1143178920 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:00.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.184+0000 7f41eeffd640 10 client.?.objecter ms_handle_connect 0x7f41f81430c0 2024-03-25T19:43:00.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.184+0000 7f41eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:00.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.186+0000 7f41eeffd640 10 client.?.objecter ms_handle_connect 0x7f41f80a98f0 2024-03-25T19:43:00.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.186+0000 7f41eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:00.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.193+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074721131 0 0) 0x7f41f813c9b0 con 0x7f41f81430c0 2024-03-25T19:43:00.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.193+0000 7f41eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.193+0000 7f41eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.193+0000 7f41eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.193+0000 7f41eeffd640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:43:00.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.193+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc004eb0 con 0x7f41f81430c0 2024-03-25T19:43:00.196 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.197+0000 7f41eeffd640 10 client.?.objecter ms_handle_connect 0x7f41f813f710 2024-03-25T19:43:00.196 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.197+0000 7f41eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:00.197 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.197+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585242608 0 0) 0x7f41f81397c0 con 0x7f41f80a98f0 2024-03-25T19:43:00.197 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.197+0000 7f41eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.197 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.197+0000 7f41eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.197 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.197+0000 7f41eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.197 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.197+0000 7f41eeffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:43:00.197 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.197+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f813c9b0 con 0x7f41f80a98f0 2024-03-25T19:43:00.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.200+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357900008 0 0) 0x7f41f8139950 con 0x7f41f813f710 2024-03-25T19:43:00.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.200+0000 7f41eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.200+0000 7f41eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.200+0000 7f41eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.200+0000 7f41eeffd640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:43:00.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.200+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f81397c0 con 0x7f41f813f710 2024-03-25T19:43:00.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.206+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1072554201 0 0) 0x7f41f81397c0 con 0x7f41f813f710 2024-03-25T19:43:00.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.207+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41f8139950 con 0x7f41f813f710 2024-03-25T19:43:00.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.207+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f41e8001830 con 0x7f41f813f710 2024-03-25T19:43:00.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.207+0000 7f41eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.207+0000 7f41eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.207+0000 7f41eeffd640 10 monclient(hunting): dump: 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:00.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.207+0000 7f41eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:00.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.208+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1061255794 0 0) 0x7f41dc004eb0 con 0x7f41f81430c0 2024-03-25T19:43:00.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.208+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41f81397c0 con 0x7f41f81430c0 2024-03-25T19:43:00.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.208+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f41f0001ec0 con 0x7f41f81430c0 2024-03-25T19:43:00.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.208+0000 7f41eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.208+0000 7f41eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:00.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.208+0000 7f41eeffd640 10 monclient(hunting): dump: 2024-03-25T19:43:00.208 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:00.209 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.209+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3088626955 0 0) 0x7f41f813c9b0 con 0x7f41f80a98f0 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.209+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41dc004eb0 con 0x7f41f80a98f0 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.209+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f41f4001970 con 0x7f41f80a98f0 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.209+0000 7f41eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.209+0000 7f41eeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.209+0000 7f41eeffd640 10 monclient(hunting): dump: 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.210 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:00.211 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:00.211 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:00.211 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:00.211 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1463653509 0 0) 0x7f41f81397c0 con 0x7f41f81430c0 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 >> v1:172.21.15.110:6790/0 conn(0x7f41f813f710 legacy=0x7f41f81dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 >> v1:172.21.15.178:6789/0 conn(0x7f41f80a98f0 legacy=0x7f41f8130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 1 monclient: found mon.a 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f8131680 con 0x7f41f81430c0 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:00.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f41eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 5 monclient: authenticate success, global_id 4338 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 10 monclient: _renew_subs 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f8131950 con 0x7f41f81430c0 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 10 monclient: _renew_subs 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f8135a30 con 0x7f41f81430c0 2024-03-25T19:43:00.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.219+0000 7f4200315640 1 librados: init done 2024-03-25T19:43:00.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.220+0000 7f4200315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.220+0000 7f4200315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.220+0000 7f4200315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:00.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.220+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f813c810 con 0x7f41f81430c0 2024-03-25T19:43:00.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.223+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41f0004e10 con 0x7f41f81430c0 2024-03-25T19:43:00.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.223+0000 7f41eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:00.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.223+0000 7f41fc8ae640 4 set_mon_vals no callback set 2024-03-25T19:43:00.512 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.512+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f41f00015d0 con 0x7f41f81430c0 2024-03-25T19:43:00.512 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.512+0000 7f41eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.512 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.512+0000 7f41eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:00.512 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.512+0000 7f41eeffd640 10 monclient: dump: 2024-03-25T19:43:00.512 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:00.512 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:00.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.513+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f41f001adf0 con 0x7f41f81430c0 2024-03-25T19:43:00.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.513+0000 7f41fe0b1640 10 monclient: get_auth_request con 0x7f41dc06e110 auth_method 0 2024-03-25T19:43:00.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.514+0000 7f41eeffd640 10 client.4338.objecter ms_handle_connect 0x7f41dc06e110 2024-03-25T19:43:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.520+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f41f0054a30 con 0x7f41f81430c0 2024-03-25T19:43:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.520+0000 7f41eeffd640 10 client.4338.objecter ms_dispatch 0x7f41f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.520+0000 7f41eeffd640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.520+0000 7f41eeffd640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.521+0000 7f41eeffd640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-25T19:43:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.521+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41f0050370 con 0x7f41f81430c0 2024-03-25T19:43:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.521+0000 7f41eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.521+0000 7f41eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:00.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.741+0000 7f4200315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.741+0000 7f4200315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.741+0000 7f4200315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:00.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.741+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41f8136fc0 con 0x7f41f81430c0 2024-03-25T19:43:00.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.780+0000 7f41eeffd640 1 -- 172.21.15.110:0/1143178920 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f41f0052470 con 0x7f41f81430c0 2024-03-25T19:43:00.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.780+0000 7f41eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.780+0000 7f41eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:00.780 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:43:00.780 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 10 librados: watch_flush enter 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 10 librados: watch_flush exit 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 10 client.4338.objecter successfully canceled tick 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 >> v1:172.21.15.178:6800/1132739058 conn(0x7f41dc06e110 legacy=0x7f41dc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 10 monclient: shutdown 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 >> v1:172.21.15.110:6789/0 conn(0x7f41f81430c0 legacy=0x7f41f81e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 shutdown_connections 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 >> 172.21.15.110:0/1143178920 conn(0x7f41f80b7ea0 msgr2=0x7f41f814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 shutdown_connections 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 1 -- 172.21.15.110:0/1143178920 wait complete. 2024-03-25T19:43:00.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:00.783+0000 7f4200315640 1 librados: shutdown 2024-03-25T19:43:00.794 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:07.600 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:07.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.722+0000 7f021a2c1640 10 monclient: get_monmap_and_config 2024-03-25T19:43:07.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.722+0000 7f021a2c1640 10 monclient: build_initial_monmap 2024-03-25T19:43:07.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.722+0000 7f021a2c1640 10 monclient: monmap: 2024-03-25T19:43:07.721 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:43:07.723061+0000 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:43:07.723061+0000 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:07.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 1 Processor -- start 2024-03-25T19:43:07.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 1 -- start start 2024-03-25T19:43:07.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 10 monclient: init 2024-03-25T19:43:07.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:07.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:07.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 10 monclient(hunting): picked mon.noname-b con 0x7f02140a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:07.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 10 monclient(hunting): picked mon.noname-c con 0x7f021413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:07.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 10 monclient(hunting): picked mon.noname-a con 0x7f02141430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:07.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214142520 con 0x7f02141430c0 2024-03-25T19:43:07.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140b28a0 con 0x7f021413f710 2024-03-25T19:43:07.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140b2a30 con 0x7f02140a98f0 2024-03-25T19:43:07.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:07.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.723+0000 7f021a2c1640 10 monclient(hunting): authenticate will time out at 4358.864644s 2024-03-25T19:43:07.736 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.736+0000 7f021885e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f02141430c0 0x7f021414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56180/0 (socket says 172.21.15.110:56180) 2024-03-25T19:43:07.736 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.736+0000 7f021885e640 1 -- 172.21.15.110:0/4066580019 learned_addr learned my addr 172.21.15.110:0/4066580019 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:07.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.752+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166276303 0 0) 0x7f0214142520 con 0x7f02141430c0 2024-03-25T19:43:07.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.752+0000 7f0212ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.752+0000 7f0212ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.752+0000 7f0212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.752+0000 7f0212ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:43:07.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.752+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01fc003520 con 0x7f02141430c0 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3865399780 0 0) 0x7f01fc003520 con 0x7f02141430c0 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 >> v1:172.21.15.110:6790/0 conn(0x7f02140a98f0 legacy=0x7f02140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 >> v1:172.21.15.178:6789/0 conn(0x7f021413f710 legacy=0x7f02141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 1 monclient: found mon.noname-a 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140b44c0 con 0x7f02141430c0 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f020c001a10 con 0x7f02141430c0 2024-03-25T19:43:07.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f0212ffd640 10 monclient: dump: 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f021a2c1640 5 monclient: authenticate success, global_id 4341 2024-03-25T19:43:07.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.762+0000 7f021a2c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:07.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.787+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f020c001be0 con 0x7f02141430c0 2024-03-25T19:43:07.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f0212ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:07.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f0212ffd640 1 -- 172.21.15.110:0/4066580019 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f020c002030 con 0x7f02141430c0 2024-03-25T19:43:07.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 4 set_mon_vals no callback set 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f0212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f0212ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f0212ffd640 10 monclient: dump: 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 10 monclient: shutdown 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 1 -- 172.21.15.110:0/4066580019 >> v1:172.21.15.110:6789/0 conn(0x7f02141430c0 legacy=0x7f021414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 1 -- 172.21.15.110:0/4066580019 shutdown_connections 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 1 -- 172.21.15.110:0/4066580019 >> 172.21.15.110:0/4066580019 conn(0x7f02140b7ea0 msgr2=0x7f02140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 1 -- 172.21.15.110:0/4066580019 shutdown_connections 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.788+0000 7f021a2c1640 1 -- 172.21.15.110:0/4066580019 wait complete. 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.789+0000 7f021a2c1640 10 monclient: build_initial_monmap 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.789+0000 7f021a2c1640 10 monclient: monmap: 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 1 librados: starting msgr at 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 1 librados: starting objecter 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 1 Processor -- start 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 1 -- start start 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 1 librados: setting wanted keys 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 1 librados: calling monclient init 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 10 monclient: init 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.790+0000 7f021a2c1640 10 monclient(hunting): picked mon.noname-a con 0x7f02140a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.791+0000 7f021a2c1640 10 monclient(hunting): picked mon.noname-b con 0x7f021413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:07.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.791+0000 7f021a2c1640 10 monclient(hunting): picked mon.noname-c con 0x7f02141430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:07.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.791+0000 7f021a2c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140b07c0 con 0x7f02140a98f0 2024-03-25T19:43:07.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.791+0000 7f021a2c1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140aef00 con 0x7f021413f710 2024-03-25T19:43:07.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.791+0000 7f021a2c1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140b0a30 con 0x7f02141430c0 2024-03-25T19:43:07.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.791+0000 7f021a2c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:07.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.791+0000 7f021a2c1640 10 monclient(hunting): authenticate will time out at 4358.931897s 2024-03-25T19:43:07.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.799+0000 7f02137fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f021413f710 0x7f02140b11c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:54390/0 (socket says 172.21.15.110:54390) 2024-03-25T19:43:07.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.799+0000 7f02137fe640 1 -- 172.21.15.110:0/475803732 learned_addr learned my addr 172.21.15.110:0/475803732 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:07.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.800+0000 7f0210ff9640 10 client.?.objecter ms_handle_connect 0x7f021413f710 2024-03-25T19:43:07.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.800+0000 7f0210ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.805+0000 7f0210ff9640 10 client.?.objecter ms_handle_connect 0x7f02141430c0 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.805+0000 7f0210ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.809+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639146318 0 0) 0x7f02140aef00 con 0x7f021413f710 2024-03-25T19:43:07.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.809+0000 7f0210ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.809+0000 7f0210ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.809+0000 7f0210ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.809+0000 7f0210ff9640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:43:07.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.809+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0200004eb0 con 0x7f021413f710 2024-03-25T19:43:07.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.809+0000 7f0210ff9640 10 client.?.objecter ms_handle_connect 0x7f02140a98f0 2024-03-25T19:43:07.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.809+0000 7f0210ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.811+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041312244 0 0) 0x7f02140b0a30 con 0x7f02141430c0 2024-03-25T19:43:07.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.811+0000 7f0210ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.811+0000 7f0210ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.811+0000 7f0210ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.811+0000 7f0210ff9640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:43:07.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.811+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02140aef00 con 0x7f02141430c0 2024-03-25T19:43:07.819 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.820+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2576121830 0 0) 0x7f0200004eb0 con 0x7f021413f710 2024-03-25T19:43:07.819 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.820+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02140b0a30 con 0x7f021413f710 2024-03-25T19:43:07.819 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.820+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0208001960 con 0x7f021413f710 2024-03-25T19:43:07.819 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.820+0000 7f0210ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.820+0000 7f0210ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.820+0000 7f0210ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:07.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.820+0000 7f0210ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:07.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.823+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691178667 0 0) 0x7f02140b07c0 con 0x7f02140a98f0 2024-03-25T19:43:07.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.823+0000 7f0210ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.823+0000 7f0210ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.823+0000 7f0210ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.823+0000 7f0210ff9640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:43:07.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.823+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0200004eb0 con 0x7f02140a98f0 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.836+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1701812913 0 0) 0x7f02140aef00 con 0x7f02141430c0 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.836+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02140b07c0 con 0x7f02141430c0 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.836+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f020c001ef0 con 0x7f02141430c0 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.836+0000 7f0210ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.836+0000 7f0210ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.836+0000 7f0210ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:07.836 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:07.837 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:07.837 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:07.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.843+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1710878148 0 0) 0x7f0200004eb0 con 0x7f02140a98f0 2024-03-25T19:43:07.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.844+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02140aef00 con 0x7f02140a98f0 2024-03-25T19:43:07.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.844+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f02040017d0 con 0x7f02140a98f0 2024-03-25T19:43:07.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.844+0000 7f0210ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.844+0000 7f0210ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:07.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.844+0000 7f0210ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:07.843 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:07.844 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:07.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3927076368 0 0) 0x7f02140b0a30 con 0x7f021413f710 2024-03-25T19:43:07.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 >> v1:172.21.15.110:6790/0 conn(0x7f02141430c0 legacy=0x7f02140af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 >> v1:172.21.15.110:6789/0 conn(0x7f02140a98f0 legacy=0x7f02140abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 1 monclient: found mon.b 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140ae880 con 0x7f021413f710 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f0210ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 10 monclient: _renew_subs 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02140aeb30 con 0x7f021413f710 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 10 monclient: _renew_subs 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02141e2e40 con 0x7f021413f710 2024-03-25T19:43:07.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.845+0000 7f021a2c1640 1 librados: init done 2024-03-25T19:43:07.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.846+0000 7f021a2c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.846+0000 7f021a2c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.846+0000 7f021a2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:07.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:07.846+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02140aed20 con 0x7f021413f710 2024-03-25T19:43:08.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.136+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02080035f0 con 0x7f021413f710 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.136+0000 7f0210ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.136+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f02080037a0 con 0x7f021413f710 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.136+0000 7f0210ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.136+0000 7f0210ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.136+0000 7f0210ff9640 10 monclient: dump: 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:08.136 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:08.137 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:08.137 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:08.137 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:08.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.136+0000 7f02127fc640 4 set_mon_vals no callback set 2024-03-25T19:43:08.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.137+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f02080039c0 con 0x7f021413f710 2024-03-25T19:43:08.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.138+0000 7f0213fff640 10 monclient: get_auth_request con 0x7f020006e110 auth_method 0 2024-03-25T19:43:08.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.138+0000 7f0210ff9640 10 client.4351.objecter ms_handle_connect 0x7f020006e110 2024-03-25T19:43:08.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.141+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f02080545f0 con 0x7f021413f710 2024-03-25T19:43:08.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.141+0000 7f0210ff9640 10 client.4351.objecter ms_dispatch 0x7f0214000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:08.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.141+0000 7f0210ff9640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:08.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.141+0000 7f0210ff9640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:08.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.141+0000 7f0210ff9640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T19:43:08.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.142+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0208050100 con 0x7f021413f710 2024-03-25T19:43:08.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.142+0000 7f0210ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:08.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.142+0000 7f0210ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:08.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.391+0000 7f021a2c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.391+0000 7f021a2c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.391+0000 7f021a2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:08.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.391+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02140af0e0 con 0x7f021413f710 2024-03-25T19:43:08.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.428+0000 7f0210ff9640 1 -- 172.21.15.110:0/475803732 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f0208052030 con 0x7f021413f710 2024-03-25T19:43:08.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.429+0000 7f0210ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.429+0000 7f0210ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:08.429 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:43:08.429 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:08.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 10 librados: watch_flush enter 2024-03-25T19:43:08.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 10 librados: watch_flush exit 2024-03-25T19:43:08.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T19:43:08.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 10 client.4351.objecter successfully canceled tick 2024-03-25T19:43:08.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 >> v1:172.21.15.178:6800/1132739058 conn(0x7f020006e110 legacy=0x7f02000705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:08.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 10 monclient: shutdown 2024-03-25T19:43:08.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:08.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 >> v1:172.21.15.178:6789/0 conn(0x7f021413f710 legacy=0x7f02140b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:08.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 shutdown_connections 2024-03-25T19:43:08.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 >> 172.21.15.110:0/475803732 conn(0x7f02140b7ea0 msgr2=0x7f0214074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:08.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 shutdown_connections 2024-03-25T19:43:08.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 1 -- 172.21.15.110:0/475803732 wait complete. 2024-03-25T19:43:08.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:08.431+0000 7f021a2c1640 1 librados: shutdown 2024-03-25T19:43:08.442 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:15.249 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:15.364 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.364+0000 7f0d46d9d640 10 monclient: get_monmap_and_config 2024-03-25T19:43:15.364 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.364+0000 7f0d46d9d640 10 monclient: build_initial_monmap 2024-03-25T19:43:15.364 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.364+0000 7f0d46d9d640 10 monclient: monmap: 2024-03-25T19:43:15.364 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:43:15.365749+0000 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:43:15.365749+0000 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:15.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.365+0000 7f0d46d9d640 1 Processor -- start 2024-03-25T19:43:15.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.365+0000 7f0d46d9d640 1 -- start start 2024-03-25T19:43:15.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.365+0000 7f0d46d9d640 10 monclient: init 2024-03-25T19:43:15.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:15.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:15.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d400a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:15.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:15.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d401430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:15.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d40142520 con 0x7f0d401430c0 2024-03-25T19:43:15.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d400b28a0 con 0x7f0d4013f710 2024-03-25T19:43:15.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d400b2a30 con 0x7f0d400a98f0 2024-03-25T19:43:15.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:15.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.366+0000 7f0d46d9d640 10 monclient(hunting): authenticate will time out at 4366.507587s 2024-03-25T19:43:15.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.382+0000 7f0d4533a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0d401430c0 0x7f0d4014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35956/0 (socket says 172.21.15.110:35956) 2024-03-25T19:43:15.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.382+0000 7f0d4533a640 1 -- 172.21.15.110:0/3035532593 learned_addr learned my addr 172.21.15.110:0/3035532593 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:15.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.401+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300279604 0 0) 0x7f0d400b2a30 con 0x7f0d400a98f0 2024-03-25T19:43:15.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.401+0000 7f0d3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.401+0000 7f0d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.401+0000 7f0d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.401+0000 7f0d3f7fe640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:43:15.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.401+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d30003520 con 0x7f0d400a98f0 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393763258 0 0) 0x7f0d40142520 con 0x7f0d401430c0 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d400b2a30 con 0x7f0d401430c0 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332509602 0 0) 0x7f0d400b28a0 con 0x7f0d4013f710 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:43:15.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.402+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d40142520 con 0x7f0d4013f710 2024-03-25T19:43:15.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 311715068 0 0) 0x7f0d400b2a30 con 0x7f0d401430c0 2024-03-25T19:43:15.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 >> v1:172.21.15.110:6790/0 conn(0x7f0d400a98f0 legacy=0x7f0d400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 >> v1:172.21.15.178:6789/0 conn(0x7f0d4013f710 legacy=0x7f0d401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 1 monclient: found mon.noname-a 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d400b44c0 con 0x7f0d401430c0 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d46d9d640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:43:15.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.410+0000 7f0d46d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.413+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0d380019b0 con 0x7f0d401430c0 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.413+0000 7f0d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.413+0000 7f0d3f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.413+0000 7f0d3f7fe640 10 monclient: dump: 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.413 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:15.414 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:15.414 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:15.414 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:15.414 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:15.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.413+0000 7f0d46d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.422+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d38001c00 con 0x7f0d401430c0 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.422+0000 7f0d3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.422+0000 7f0d3f7fe640 1 -- 172.21.15.110:0/3035532593 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0d38001f90 con 0x7f0d401430c0 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.422+0000 7f0d46d9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.422+0000 7f0d46d9d640 4 set_mon_vals no callback set 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.422+0000 7f0d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.422+0000 7f0d3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.422+0000 7f0d3f7fe640 10 monclient: dump: 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.423+0000 7f0d46d9d640 10 monclient: shutdown 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.423+0000 7f0d46d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.423+0000 7f0d46d9d640 1 -- 172.21.15.110:0/3035532593 >> v1:172.21.15.110:6789/0 conn(0x7f0d401430c0 legacy=0x7f0d4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.423+0000 7f0d46d9d640 1 -- 172.21.15.110:0/3035532593 shutdown_connections 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.423+0000 7f0d46d9d640 1 -- 172.21.15.110:0/3035532593 >> 172.21.15.110:0/3035532593 conn(0x7f0d400b7ea0 msgr2=0x7f0d400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.423+0000 7f0d46d9d640 1 -- 172.21.15.110:0/3035532593 shutdown_connections 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.423+0000 7f0d46d9d640 1 -- 172.21.15.110:0/3035532593 wait complete. 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.424+0000 7f0d46d9d640 10 monclient: build_initial_monmap 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.424+0000 7f0d46d9d640 10 monclient: monmap: 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:15.425 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.424+0000 7f0d46d9d640 1 librados: starting msgr at 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.424+0000 7f0d46d9d640 1 librados: starting objecter 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.425+0000 7f0d46d9d640 1 Processor -- start 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.425+0000 7f0d46d9d640 1 -- start start 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.425+0000 7f0d46d9d640 1 librados: setting wanted keys 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.425+0000 7f0d46d9d640 1 librados: calling monclient init 2024-03-25T19:43:15.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.425+0000 7f0d46d9d640 10 monclient: init 2024-03-25T19:43:15.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:15.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:15.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0d400a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:15.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0d4013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:15.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0d401430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:15.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d401da880 con 0x7f0d4013f710 2024-03-25T19:43:15.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d401db700 con 0x7f0d401430c0 2024-03-25T19:43:15.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d400abc30 con 0x7f0d400a98f0 2024-03-25T19:43:15.429 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:15.429 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.426+0000 7f0d46d9d640 10 monclient(hunting): authenticate will time out at 4366.567042s 2024-03-25T19:43:15.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.434+0000 7f0d44b39640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f0d400a98f0 0x7f0d401daff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:37042/0 (socket says 172.21.15.110:37042) 2024-03-25T19:43:15.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.434+0000 7f0d44b39640 1 -- 172.21.15.110:0/883123535 learned_addr learned my addr 172.21.15.110:0/883123535 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:15.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.436+0000 7f0d3d7fa640 10 client.?.objecter ms_handle_connect 0x7f0d4013f710 2024-03-25T19:43:15.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.436+0000 7f0d3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.439+0000 7f0d3d7fa640 10 client.?.objecter ms_handle_connect 0x7f0d400a98f0 2024-03-25T19:43:15.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.439+0000 7f0d3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.444 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.444+0000 7f0d3d7fa640 10 client.?.objecter ms_handle_connect 0x7f0d401430c0 2024-03-25T19:43:15.444 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.444+0000 7f0d3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.453+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153673077 0 0) 0x7f0d400abc30 con 0x7f0d400a98f0 2024-03-25T19:43:15.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.453+0000 7f0d3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.453+0000 7f0d3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.453+0000 7f0d3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.453+0000 7f0d3d7fa640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:43:15.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.453+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d24004eb0 con 0x7f0d400a98f0 2024-03-25T19:43:15.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.455+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891760507 0 0) 0x7f0d401db700 con 0x7f0d401430c0 2024-03-25T19:43:15.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.455+0000 7f0d3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.455+0000 7f0d3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.455+0000 7f0d3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.455+0000 7f0d3d7fa640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:43:15.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.455+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d400abc30 con 0x7f0d401430c0 2024-03-25T19:43:15.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.460+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185822905 0 0) 0x7f0d401da880 con 0x7f0d4013f710 2024-03-25T19:43:15.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.460+0000 7f0d3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.460+0000 7f0d3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.460+0000 7f0d3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.460+0000 7f0d3d7fa640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:43:15.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.460+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d401db700 con 0x7f0d4013f710 2024-03-25T19:43:15.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.465+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793493374 0 0) 0x7f0d400abc30 con 0x7f0d401430c0 2024-03-25T19:43:15.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.465+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d401da880 con 0x7f0d401430c0 2024-03-25T19:43:15.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.465+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0d38001f10 con 0x7f0d401430c0 2024-03-25T19:43:15.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.465+0000 7f0d3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.465+0000 7f0d3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:15.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.465+0000 7f0d3d7fa640 10 monclient(hunting): dump: 2024-03-25T19:43:15.466 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:15.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.465+0000 7f0d3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:15.475 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.476+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 995399936 0 0) 0x7f0d24004eb0 con 0x7f0d400a98f0 2024-03-25T19:43:15.475 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.476+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d400abc30 con 0x7f0d400a98f0 2024-03-25T19:43:15.475 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.476+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0d34001990 con 0x7f0d400a98f0 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.476+0000 7f0d3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.476+0000 7f0d3d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.476+0000 7f0d3d7fa640 10 monclient(hunting): dump: 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:15.476 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:15.477 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2040250508 0 0) 0x7f0d401da880 con 0x7f0d401430c0 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 >> v1:172.21.15.110:6790/0 conn(0x7f0d400a98f0 legacy=0x7f0d401daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 >> v1:172.21.15.110:6789/0 conn(0x7f0d4013f710 legacy=0x7f0d401381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 1 monclient: found mon.b 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d401daad0 con 0x7f0d401430c0 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 5 monclient: authenticate success, global_id 4357 2024-03-25T19:43:15.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 10 monclient: _renew_subs 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d40137930 con 0x7f0d401430c0 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 10 monclient: _renew_subs 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d40138a90 con 0x7f0d401430c0 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.477+0000 7f0d46d9d640 1 librados: init done 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.478+0000 7f0d46d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.478+0000 7f0d46d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.478+0000 7f0d46d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:15.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.478+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d400aba70 con 0x7f0d401430c0 2024-03-25T19:43:15.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.763+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d38004e10 con 0x7f0d401430c0 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.763+0000 7f0d3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.763+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0d38005120 con 0x7f0d401430c0 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.763+0000 7f0d3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.763+0000 7f0d3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.763+0000 7f0d3d7fa640 10 monclient: dump: 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:15.763 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:15.764 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:15.764 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:15.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.763+0000 7f0d3effd640 4 set_mon_vals no callback set 2024-03-25T19:43:15.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.764+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f0d3801b0f0 con 0x7f0d401430c0 2024-03-25T19:43:15.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.764+0000 7f0d44b39640 10 monclient: get_auth_request con 0x7f0d2406df60 auth_method 0 2024-03-25T19:43:15.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.764+0000 7f0d3d7fa640 10 client.4357.objecter ms_handle_connect 0x7f0d2406df60 2024-03-25T19:43:15.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.769+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f0d38054ba0 con 0x7f0d401430c0 2024-03-25T19:43:15.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.769+0000 7f0d3d7fa640 10 client.4357.objecter ms_dispatch 0x7f0d40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:15.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.769+0000 7f0d3d7fa640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:15.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.769+0000 7f0d3d7fa640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:15.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.769+0000 7f0d3d7fa640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-25T19:43:15.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.770+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d380504e0 con 0x7f0d401430c0 2024-03-25T19:43:15.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.770+0000 7f0d3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:15.770+0000 7f0d3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:16.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.008+0000 7f0d46d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:16.008 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.008+0000 7f0d46d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:16.008 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.008+0000 7f0d46d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:16.008 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.008+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d40074cc0 con 0x7f0d401430c0 2024-03-25T19:43:16.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.037+0000 7f0d3d7fa640 1 -- 172.21.15.110:0/883123535 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f0d380525e0 con 0x7f0d401430c0 2024-03-25T19:43:16.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.037+0000 7f0d3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:16.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.037+0000 7f0d3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:16.037 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:43:16.037 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:16.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 10 librados: watch_flush enter 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 10 librados: watch_flush exit 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 10 client.4357.objecter successfully canceled tick 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 >> v1:172.21.15.178:6800/1132739058 conn(0x7f0d2406df60 legacy=0x7f0d24070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 10 monclient: shutdown 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 >> v1:172.21.15.178:6789/0 conn(0x7f0d401430c0 legacy=0x7f0d401e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 shutdown_connections 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.039+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 >> 172.21.15.110:0/883123535 conn(0x7f0d400b7ea0 msgr2=0x7f0d40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.040+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 shutdown_connections 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.040+0000 7f0d46d9d640 1 -- 172.21.15.110:0/883123535 wait complete. 2024-03-25T19:43:16.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:16.040+0000 7f0d46d9d640 1 librados: shutdown 2024-03-25T19:43:16.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:17.310 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:24.594 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:17.314 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:24.569 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:17.319 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:24.604 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:17.323 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:24.577 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:17.340 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:24.625 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:17.365 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:24.620 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:17.378 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:24.662 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:17.393 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:24.678 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:17.421 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:24.706 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:22.857 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.971+0000 7fa1063d6640 10 monclient: get_monmap_and_config 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.971+0000 7fa1063d6640 10 monclient: build_initial_monmap 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.971+0000 7fa1063d6640 10 monclient: monmap: 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:43:22.972553+0000 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:43:22.972553+0000 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:22.971 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:43:22.972 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:43:22.972 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.972+0000 7fa1063d6640 1 Processor -- start 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.972+0000 7fa1063d6640 1 -- start start 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.972+0000 7fa1063d6640 10 monclient: init 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa1000a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa10013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa1001430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:22.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa100142520 con 0x7fa10013f710 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000b28a0 con 0x7fa1001430c0 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000b2a30 con 0x7fa1000a98f0 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.973+0000 7fa1063d6640 10 monclient(hunting): authenticate will time out at 4374.114426s 2024-03-25T19:43:22.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.980+0000 7fa0fffff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fa1000a98f0 0x7fa1000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40414/0 (socket says 172.21.15.110:40414) 2024-03-25T19:43:22.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.981+0000 7fa0fffff640 1 -- 172.21.15.110:0/1939751146 learned_addr learned my addr 172.21.15.110:0/1939751146 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:22.989 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.990+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312978723 0 0) 0x7fa1000b2a30 con 0x7fa1000a98f0 2024-03-25T19:43:22.989 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.990+0000 7fa0ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.989 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.990+0000 7fa0ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.989 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.990+0000 7fa0ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.990+0000 7fa0ff7fe640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:43:22.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.990+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ec003520 con 0x7fa1000a98f0 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.996+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096738568 0 0) 0x7fa1000b28a0 con 0x7fa1001430c0 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.996+0000 7fa0ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.996+0000 7fa0ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.996+0000 7fa0ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.996+0000 7fa0ff7fe640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:22.996+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1000b2a30 con 0x7fa1001430c0 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.000+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3120995697 0 0) 0x7fa0ec003520 con 0x7fa1000a98f0 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 >> v1:172.21.15.178:6789/0 conn(0x7fa1001430c0 legacy=0x7fa10014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 >> v1:172.21.15.110:6789/0 conn(0x7fa10013f710 legacy=0x7fa1001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 1 monclient: found mon.noname-b 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1000b44c0 con 0x7fa1000a98f0 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa0f0001690 con 0x7fa1000a98f0 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa0ff7fe640 10 monclient: dump: 2024-03-25T19:43:23.001 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa1063d6640 5 monclient: authenticate success, global_id 4355 2024-03-25T19:43:23.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.001+0000 7fa1063d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.019+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f00017f0 con 0x7fa1000a98f0 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa0ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa0ff7fe640 1 -- 172.21.15.110:0/1939751146 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa0f0001c00 con 0x7fa1000a98f0 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa1063d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa1063d6640 4 set_mon_vals no callback set 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa0ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa0ff7fe640 10 monclient: dump: 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:23.020 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:23.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa1063d6640 10 monclient: shutdown 2024-03-25T19:43:23.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa1063d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:23.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.020+0000 7fa1063d6640 1 -- 172.21.15.110:0/1939751146 >> v1:172.21.15.110:6790/0 conn(0x7fa1000a98f0 legacy=0x7fa1000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.021+0000 7fa1063d6640 1 -- 172.21.15.110:0/1939751146 shutdown_connections 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.021+0000 7fa1063d6640 1 -- 172.21.15.110:0/1939751146 >> 172.21.15.110:0/1939751146 conn(0x7fa1000b7ea0 msgr2=0x7fa1000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.021+0000 7fa1063d6640 1 -- 172.21.15.110:0/1939751146 shutdown_connections 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.021+0000 7fa1063d6640 1 -- 172.21.15.110:0/1939751146 wait complete. 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 10 monclient: build_initial_monmap 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 10 monclient: monmap: 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:43:23.023 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:43:23.023 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:23.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 1 librados: starting msgr at 2024-03-25T19:43:23.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 1 librados: starting objecter 2024-03-25T19:43:23.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 1 Processor -- start 2024-03-25T19:43:23.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 1 -- start start 2024-03-25T19:43:23.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 1 librados: setting wanted keys 2024-03-25T19:43:23.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 1 librados: calling monclient init 2024-03-25T19:43:23.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.022+0000 7fa1063d6640 10 monclient: init 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa1000a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa10013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa1001430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10013a820 con 0x7fa1000a98f0 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10013b3f0 con 0x7fa10013f710 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10013b6d0 con 0x7fa1001430c0 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:23.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.023+0000 7fa1063d6640 10 monclient(hunting): authenticate will time out at 4374.164377s 2024-03-25T19:43:23.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.028+0000 7fa104973640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fa1001430c0 0x7fa1001e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40422/0 (socket says 172.21.15.110:40422) 2024-03-25T19:43:23.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.028+0000 7fa104973640 1 -- 172.21.15.110:0/301301190 learned_addr learned my addr 172.21.15.110:0/301301190 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:23.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.031+0000 7fa0fd7fa640 10 client.?.objecter ms_handle_connect 0x7fa1000a98f0 2024-03-25T19:43:23.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.032+0000 7fa0fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:23.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.032+0000 7fa0fd7fa640 10 client.?.objecter ms_handle_connect 0x7fa1001430c0 2024-03-25T19:43:23.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.032+0000 7fa0fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:23.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.035+0000 7fa0fd7fa640 10 client.?.objecter ms_handle_connect 0x7fa10013f710 2024-03-25T19:43:23.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.035+0000 7fa0fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:23.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.047+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252530278 0 0) 0x7fa10013b6d0 con 0x7fa1001430c0 2024-03-25T19:43:23.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.047+0000 7fa0fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:23.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.047+0000 7fa0fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:23.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.047+0000 7fa0fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:23.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.047+0000 7fa0fd7fa640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:43:23.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.047+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e0004eb0 con 0x7fa1001430c0 2024-03-25T19:43:23.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.048+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39143185 0 0) 0x7fa10013b3f0 con 0x7fa10013f710 2024-03-25T19:43:23.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.048+0000 7fa0fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:23.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.048+0000 7fa0fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:23.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.048+0000 7fa0fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:23.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.048+0000 7fa0fd7fa640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:43:23.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.048+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa10013b6d0 con 0x7fa10013f710 2024-03-25T19:43:23.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.056+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398847513 0 0) 0x7fa10013a820 con 0x7fa1000a98f0 2024-03-25T19:43:23.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.056+0000 7fa0fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:23.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.056+0000 7fa0fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:23.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.056+0000 7fa0fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:23.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.056+0000 7fa0fd7fa640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:43:23.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.056+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa10013b3f0 con 0x7fa1000a98f0 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.069+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41032548 0 0) 0x7fa10013b3f0 con 0x7fa1000a98f0 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.069+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa10013a820 con 0x7fa1000a98f0 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.069+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa0f0001b00 con 0x7fa1000a98f0 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.069+0000 7fa0fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.069+0000 7fa0fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.069+0000 7fa0fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.069 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:23.070 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:23.070 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:23.070 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:23.070 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:23.070 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.069+0000 7fa0fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.078+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 322075040 0 0) 0x7fa10013b6d0 con 0x7fa10013f710 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.078+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa10013b3f0 con 0x7fa10013f710 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.078+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa0e8001510 con 0x7fa10013f710 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.078+0000 7fa0fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.078+0000 7fa0fd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.078+0000 7fa0fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.078 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.079 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:23.079 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:23.079 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:23.079 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:23.079 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:23.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.079+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478673912 0 0) 0x7fa0e0004eb0 con 0x7fa1001430c0 2024-03-25T19:43:23.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.079+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa10013b6d0 con 0x7fa1001430c0 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.079+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa0f8001d20 con 0x7fa1001430c0 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.079+0000 7fa0fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.079+0000 7fa0fd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.079+0000 7fa0fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:23.080 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1163745031 0 0) 0x7fa10013a820 con 0x7fa1000a98f0 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 >> v1:172.21.15.110:6790/0 conn(0x7fa1001430c0 legacy=0x7fa1001e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 >> v1:172.21.15.178:6789/0 conn(0x7fa10013f710 legacy=0x7fa1001df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 1 monclient: found mon.a 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10013b9b0 con 0x7fa1000a98f0 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa0fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 5 monclient: authenticate success, global_id 4353 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 10 monclient: _renew_subs 2024-03-25T19:43:23.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:23.085 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa100138fd0 con 0x7fa1000a98f0 2024-03-25T19:43:23.085 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:23.085 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 10 monclient: _renew_subs 2024-03-25T19:43:23.085 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:23.085 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10013bdd0 con 0x7fa1000a98f0 2024-03-25T19:43:23.085 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.084+0000 7fa1063d6640 1 librados: init done 2024-03-25T19:43:23.085 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.085+0000 7fa1063d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:23.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.085+0000 7fa1063d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:23.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.085+0000 7fa1063d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:23.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.085+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1000b3ef0 con 0x7fa1000a98f0 2024-03-25T19:43:23.394 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.394+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f0001c80 con 0x7fa1000a98f0 2024-03-25T19:43:23.394 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.394+0000 7fa0fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:23.394 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.394+0000 7fa0feffd640 4 set_mon_vals no callback set 2024-03-25T19:43:23.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.395+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa0f0004c30 con 0x7fa1000a98f0 2024-03-25T19:43:23.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.395+0000 7fa0fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.395+0000 7fa0fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:23.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.395+0000 7fa0fd7fa640 10 monclient: dump: 2024-03-25T19:43:23.395 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:23.395 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:23.395 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.395+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fa0f001ac00 con 0x7fa1000a98f0 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.396+0000 7fa0f7fff640 10 monclient: get_auth_request con 0x7fa0e006e0b0 auth_method 0 2024-03-25T19:43:23.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.396+0000 7fa0fd7fa640 10 client.4353.objecter ms_handle_connect 0x7fa0e006e0b0 2024-03-25T19:43:23.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.402+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fa0f0054840 con 0x7fa1000a98f0 2024-03-25T19:43:23.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.402+0000 7fa0fd7fa640 10 client.4353.objecter ms_dispatch 0x7fa100000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:23.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.402+0000 7fa0fd7fa640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:23.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.402+0000 7fa0fd7fa640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:23.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.402+0000 7fa0fd7fa640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T19:43:23.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.402+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0f0050350 con 0x7fa1000a98f0 2024-03-25T19:43:23.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.402+0000 7fa0fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:23.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.402+0000 7fa0fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:23.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.657+0000 7fa1063d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.657+0000 7fa1063d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.657+0000 7fa1063d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:23.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.657+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1000627b0 con 0x7fa1000a98f0 2024-03-25T19:43:23.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.696+0000 7fa0fd7fa640 1 -- 172.21.15.110:0/301301190 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fa0f0052280 con 0x7fa1000a98f0 2024-03-25T19:43:23.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.697+0000 7fa0fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.697+0000 7fa0fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:23.696 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:43:23.697 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:23.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.698+0000 7fa1063d6640 10 librados: watch_flush enter 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.698+0000 7fa1063d6640 10 librados: watch_flush exit 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.698+0000 7fa1063d6640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.698+0000 7fa1063d6640 10 client.4353.objecter successfully canceled tick 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 >> v1:172.21.15.178:6800/1132739058 conn(0x7fa0e006e0b0 legacy=0x7fa0e0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 10 monclient: shutdown 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 >> v1:172.21.15.110:6789/0 conn(0x7fa1000a98f0 legacy=0x7fa1000b7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 shutdown_connections 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 >> 172.21.15.110:0/301301190 conn(0x7fa1000b7ea0 msgr2=0x7fa100074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 shutdown_connections 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 1 -- 172.21.15.110:0/301301190 wait complete. 2024-03-25T19:43:23.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:23.699+0000 7fa1063d6640 1 librados: shutdown 2024-03-25T19:43:23.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:30.518 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:30.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.631+0000 7f46fd196640 10 monclient: get_monmap_and_config 2024-03-25T19:43:30.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.631+0000 7f46fd196640 10 monclient: build_initial_monmap 2024-03-25T19:43:30.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.631+0000 7f46fd196640 10 monclient: monmap: 2024-03-25T19:43:30.631 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:43:30.632843+0000 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:43:30.632843+0000 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:43:30.632 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:30.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.632+0000 7f46fd196640 1 Processor -- start 2024-03-25T19:43:30.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.632+0000 7f46fd196640 1 -- start start 2024-03-25T19:43:30.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.632+0000 7f46fd196640 10 monclient: init 2024-03-25T19:43:30.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:30.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 10 monclient(hunting): picked mon.noname-a con 0x7f46f806c360 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 10 monclient(hunting): picked mon.noname-c con 0x7f46f806d5e0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 10 monclient(hunting): picked mon.noname-b con 0x7f46f80af9a0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80add00 con 0x7f46f806c360 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80adfe0 con 0x7f46f806d5e0 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f806d190 con 0x7f46f80af9a0 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.633+0000 7f46fd196640 10 monclient(hunting): authenticate will time out at 4381.774380s 2024-03-25T19:43:30.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.643+0000 7f46f6d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f46f806c360 0x7f46f80b0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:47366/0 (socket says 172.21.15.110:47366) 2024-03-25T19:43:30.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.643+0000 7f46f6d9d640 1 -- 172.21.15.110:0/2402008839 learned_addr learned my addr 172.21.15.110:0/2402008839 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.660+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417137100 0 0) 0x7f46f80add00 con 0x7f46f806c360 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.660+0000 7f46f659c640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.660+0000 7f46f659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.660+0000 7f46f659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.660+0000 7f46f659c640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.660+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e4003520 con 0x7f46f806c360 2024-03-25T19:43:30.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.666+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379605065 0 0) 0x7f46f806d190 con 0x7f46f80af9a0 2024-03-25T19:43:30.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.666+0000 7f46f659c640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.666+0000 7f46f659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.666+0000 7f46f659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.666+0000 7f46f659c640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:43:30.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.666+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f80add00 con 0x7f46f80af9a0 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.679+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 468626939 0 0) 0x7f46e4003520 con 0x7f46f806c360 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 >> v1:172.21.15.110:6790/0 conn(0x7f46f80af9a0 legacy=0x7f46f80aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 >> v1:172.21.15.178:6789/0 conn(0x7f46f806d5e0 legacy=0x7f46f80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 10 monclient: _finish_hunting 0 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 1 monclient: found mon.noname-a 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80b0d10 con 0x7f46f806c360 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 10 monclient: _finish_auth 0 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 10 monclient: _check_auth_tickets 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f46e8001690 con 0x7f46f806c360 2024-03-25T19:43:30.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46fd196640 5 monclient: authenticate success, global_id 4356 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46fd196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46f659c640 10 monclient: dump: 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:30.681 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:30.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.680+0000 7f46fd196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.756+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46e80017f0 con 0x7f46f806c360 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.756+0000 7f46f659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.756+0000 7f46f659c640 1 -- 172.21.15.110:0/2402008839 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f46e8001b80 con 0x7f46f806c360 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.756+0000 7f46f659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.756+0000 7f46f659c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.756+0000 7f46f659c640 10 monclient: dump: 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:30.756 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:30.757 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.757 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:30.757 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:30.757 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:30.757 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:30.757 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:30.757 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.756+0000 7f46fd196640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:30.757 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.756+0000 7f46fd196640 4 set_mon_vals no callback set 2024-03-25T19:43:30.759 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.757+0000 7f46fd196640 10 monclient: shutdown 2024-03-25T19:43:30.759 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.757+0000 7f46fd196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:30.759 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.757+0000 7f46fd196640 1 -- 172.21.15.110:0/2402008839 >> v1:172.21.15.110:6789/0 conn(0x7f46f806c360 legacy=0x7f46f80b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.759 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.757+0000 7f46fd196640 1 -- 172.21.15.110:0/2402008839 shutdown_connections 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.757+0000 7f46fd196640 1 -- 172.21.15.110:0/2402008839 >> 172.21.15.110:0/2402008839 conn(0x7f46f80b5c00 msgr2=0x7f46f80b6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.757+0000 7f46fd196640 1 -- 172.21.15.110:0/2402008839 shutdown_connections 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.757+0000 7f46fd196640 1 -- 172.21.15.110:0/2402008839 wait complete. 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 10 monclient: build_initial_monmap 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 10 monclient: monmap: 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:43:30.760 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 1 librados: starting msgr at 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 1 librados: starting objecter 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 1 Processor -- start 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 1 -- start start 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 1 librados: setting wanted keys 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 1 librados: calling monclient init 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.758+0000 7f46fd196640 10 monclient: init 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 10 monclient(hunting): picked mon.noname-c con 0x7f46f806c360 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 10 monclient(hunting): picked mon.noname-a con 0x7f46f806d5e0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 10 monclient(hunting): picked mon.noname-b con 0x7f46f80af9a0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f8143570 con 0x7f46f806d5e0 2024-03-25T19:43:30.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f8143850 con 0x7f46f80af9a0 2024-03-25T19:43:30.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f81426a0 con 0x7f46f806c360 2024-03-25T19:43:30.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:30.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.759+0000 7f46fd196640 10 monclient(hunting): authenticate will time out at 4381.900571s 2024-03-25T19:43:30.770 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.771+0000 7f46f6d9d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f46f806c360 0x7f46f8141bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:45292/0 (socket says 172.21.15.110:45292) 2024-03-25T19:43:30.770 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.771+0000 7f46f6d9d640 1 -- 172.21.15.110:0/2521748745 learned_addr learned my addr 172.21.15.110:0/2521748745 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:30.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.773+0000 7f46ef7fe640 10 client.?.objecter ms_handle_connect 0x7f46f806c360 2024-03-25T19:43:30.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.773+0000 7f46ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.778+0000 7f46ef7fe640 10 client.?.objecter ms_handle_connect 0x7f46f80af9a0 2024-03-25T19:43:30.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.778+0000 7f46ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.778+0000 7f46ef7fe640 10 client.?.objecter ms_handle_connect 0x7f46f806d5e0 2024-03-25T19:43:30.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.778+0000 7f46ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.783+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817933067 0 0) 0x7f46f81426a0 con 0x7f46f806c360 2024-03-25T19:43:30.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.783+0000 7f46ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.783+0000 7f46ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.783+0000 7f46ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.783+0000 7f46ef7fe640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:43:30.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.783+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d8004eb0 con 0x7f46f806c360 2024-03-25T19:43:30.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.788+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159928133 0 0) 0x7f46f8143570 con 0x7f46f806d5e0 2024-03-25T19:43:30.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.788+0000 7f46ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.788+0000 7f46ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.788+0000 7f46ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.788+0000 7f46ef7fe640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:43:30.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.788+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f81426a0 con 0x7f46f806d5e0 2024-03-25T19:43:30.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.792+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614636922 0 0) 0x7f46f8143850 con 0x7f46f80af9a0 2024-03-25T19:43:30.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.792+0000 7f46ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.792+0000 7f46ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.792+0000 7f46ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.792+0000 7f46ef7fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:43:30.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.792+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f8143570 con 0x7f46f80af9a0 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.806+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2654063668 0 0) 0x7f46f8143570 con 0x7f46f80af9a0 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.806+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f8143850 con 0x7f46f80af9a0 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.806+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f46f0001bd0 con 0x7f46f80af9a0 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.806+0000 7f46ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.806+0000 7f46ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.806+0000 7f46ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:30.806 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:30.807 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:30.807 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:30.807 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:30.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.806+0000 7f46ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:30.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.812+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038430628 0 0) 0x7f46d8004eb0 con 0x7f46f806c360 2024-03-25T19:43:30.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.812+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f8143570 con 0x7f46f806c360 2024-03-25T19:43:30.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.812+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f46e8001b00 con 0x7f46f806c360 2024-03-25T19:43:30.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.812+0000 7f46ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.812+0000 7f46ef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:30.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.812+0000 7f46ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:30.813 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:30.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3269912337 0 0) 0x7f46f8143850 con 0x7f46f80af9a0 2024-03-25T19:43:30.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 >> v1:172.21.15.110:6790/0 conn(0x7f46f806c360 legacy=0x7f46f8141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 >> v1:172.21.15.110:6789/0 conn(0x7f46f806d5e0 legacy=0x7f46f8135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 1 monclient: found mon.b 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f81416b0 con 0x7f46f80af9a0 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 10 monclient: _renew_subs 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f81422e0 con 0x7f46f80af9a0 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:30.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 10 monclient: _renew_subs 2024-03-25T19:43:30.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:30.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f8136730 con 0x7f46f80af9a0 2024-03-25T19:43:30.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.831+0000 7f46fd196640 1 librados: init done 2024-03-25T19:43:30.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.833+0000 7f46fd196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.833+0000 7f46fd196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.833+0000 7f46fd196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:30.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:30.833+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f80627b0 con 0x7f46f80af9a0 2024-03-25T19:43:31.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.112+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46f0001fe0 con 0x7f46f80af9a0 2024-03-25T19:43:31.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.112+0000 7f46ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:31.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.112+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f46f0003b10 con 0x7f46f80af9a0 2024-03-25T19:43:31.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.112+0000 7f46ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:31.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.112+0000 7f46ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:31.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.112+0000 7f46ef7fe640 10 monclient: dump: 2024-03-25T19:43:31.112 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:31.112 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:31.112 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:31.112 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:31.113 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:31.113 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:31.113 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:31.113 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:31.113 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:31.113 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:31.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.112+0000 7f46f5d9b640 4 set_mon_vals no callback set 2024-03-25T19:43:31.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.113+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f46f0019ae0 con 0x7f46f80af9a0 2024-03-25T19:43:31.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.114+0000 7f46f6d9d640 10 monclient: get_auth_request con 0x7f46d806df20 auth_method 0 2024-03-25T19:43:31.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.114+0000 7f46ef7fe640 10 client.4366.objecter ms_handle_connect 0x7f46d806df20 2024-03-25T19:43:31.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.120+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f46f0053780 con 0x7f46f80af9a0 2024-03-25T19:43:31.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.120+0000 7f46ef7fe640 10 client.4366.objecter ms_dispatch 0x7f46f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:31.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.120+0000 7f46ef7fe640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:31.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.120+0000 7f46ef7fe640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:31.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.120+0000 7f46ef7fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:43:31.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.121+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46f004f0c0 con 0x7f46f80af9a0 2024-03-25T19:43:31.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.121+0000 7f46ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:31.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.121+0000 7f46ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:31.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.377+0000 7f46fd196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.377+0000 7f46fd196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.377+0000 7f46fd196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:31.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.377+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46f80b0f90 con 0x7f46f80af9a0 2024-03-25T19:43:31.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.414+0000 7f46ef7fe640 1 -- 172.21.15.110:0/2521748745 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f46f00511c0 con 0x7f46f80af9a0 2024-03-25T19:43:31.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.414+0000 7f46ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.414+0000 7f46ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:31.414 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:43:31.414 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:31.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 10 librados: watch_flush enter 2024-03-25T19:43:31.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 10 librados: watch_flush exit 2024-03-25T19:43:31.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:43:31.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:43:31.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 >> v1:172.21.15.178:6800/1132739058 conn(0x7f46d806df20 legacy=0x7f46d80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:31.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 10 monclient: shutdown 2024-03-25T19:43:31.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:31.417 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 >> v1:172.21.15.178:6789/0 conn(0x7f46f80af9a0 legacy=0x7f46f8136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:31.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 shutdown_connections 2024-03-25T19:43:31.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 >> 172.21.15.110:0/2521748745 conn(0x7f46f80b5c00 msgr2=0x7f46f80b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:31.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 shutdown_connections 2024-03-25T19:43:31.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 1 -- 172.21.15.110:0/2521748745 wait complete. 2024-03-25T19:43:31.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:31.417+0000 7f46fd196640 1 librados: shutdown 2024-03-25T19:43:31.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:37.362 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:44.616 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.364 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:44.649 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.366 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:44.621 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.372 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:44.656 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.380 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:03:44.635 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.397 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:44.681 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.417 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:44.701 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.428 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:44.712 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.464 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:44.748 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:38.235 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:38.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.346+0000 7fe8731f6640 10 monclient: get_monmap_and_config 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.346+0000 7fe8731f6640 10 monclient: build_initial_monmap 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.346+0000 7fe8731f6640 10 monclient: monmap: 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:43:38.347326+0000 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:43:38.347326+0000 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:43:38.347 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 1 Processor -- start 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 1 -- start start 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 10 monclient: init 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe86c0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe86c13f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe86c1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c142520 con 0x7fe86c1430c0 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c0b28a0 con 0x7fe86c0a98f0 2024-03-25T19:43:38.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c0b2a30 con 0x7fe86c13f710 2024-03-25T19:43:38.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:38.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.347+0000 7fe8731f6640 10 monclient(hunting): authenticate will time out at 4389.488797s 2024-03-25T19:43:38.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.348+0000 7fe870f92640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fe86c0a98f0 0x7fe86c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:49714/0 (socket says 172.21.15.110:49714) 2024-03-25T19:43:38.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.348+0000 7fe870f92640 1 -- 172.21.15.110:0/4179360258 learned_addr learned my addr 172.21.15.110:0/4179360258 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:38.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.449+0000 7fe86b7fe640 1 -- 172.21.15.110:0/4179360258 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814945993 0 0) 0x7fe86c142520 con 0x7fe86c1430c0 2024-03-25T19:43:38.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.449+0000 7fe86b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.449+0000 7fe86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.449 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.449+0000 7fe86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.449 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.449+0000 7fe86b7fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:43:38.449 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.449+0000 7fe86b7fe640 1 -- 172.21.15.110:0/4179360258 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe85c003520 con 0x7fe86c1430c0 2024-03-25T19:43:38.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 1 -- 172.21.15.110:0/4179360258 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1885680815 0 0) 0x7fe85c003520 con 0x7fe86c1430c0 2024-03-25T19:43:38.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 1 -- 172.21.15.110:0/4179360258 >> v1:172.21.15.110:6790/0 conn(0x7fe86c13f710 legacy=0x7fe86c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 1 -- 172.21.15.110:0/4179360258 >> v1:172.21.15.178:6789/0 conn(0x7fe86c0a98f0 legacy=0x7fe86c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 1 -- 172.21.15.110:0/4179360258 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c0b44c0 con 0x7fe86c1430c0 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe86b7fe640 1 -- 172.21.15.110:0/4179360258 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe864001a10 con 0x7fe86c1430c0 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.462+0000 7fe8731f6640 5 monclient: authenticate success, global_id 4362 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.463+0000 7fe8731f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.463+0000 7fe86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.463+0000 7fe86b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.463+0000 7fe86b7fe640 10 monclient: dump: 2024-03-25T19:43:38.463 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:38.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.463+0000 7fe8731f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:38.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.468+0000 7fe86b7fe640 1 -- 172.21.15.110:0/4179360258 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe864001b70 con 0x7fe86c1430c0 2024-03-25T19:43:38.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.468+0000 7fe86b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:38.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.468+0000 7fe8731f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:38.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.468+0000 7fe8731f6640 4 set_mon_vals no callback set 2024-03-25T19:43:38.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.469+0000 7fe8731f6640 10 monclient: shutdown 2024-03-25T19:43:38.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.469+0000 7fe8731f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:38.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.469+0000 7fe8731f6640 1 -- 172.21.15.110:0/4179360258 >> v1:172.21.15.110:6789/0 conn(0x7fe86c1430c0 legacy=0x7fe86c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.469+0000 7fe8731f6640 1 -- 172.21.15.110:0/4179360258 shutdown_connections 2024-03-25T19:43:38.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.469+0000 7fe8731f6640 1 -- 172.21.15.110:0/4179360258 >> 172.21.15.110:0/4179360258 conn(0x7fe86c0b7ea0 msgr2=0x7fe86c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:38.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.469+0000 7fe8731f6640 1 -- 172.21.15.110:0/4179360258 shutdown_connections 2024-03-25T19:43:38.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.469+0000 7fe8731f6640 1 -- 172.21.15.110:0/4179360258 wait complete. 2024-03-25T19:43:38.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 10 monclient: build_initial_monmap 2024-03-25T19:43:38.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 10 monclient: monmap: 2024-03-25T19:43:38.470 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:38.470 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:38.470 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:43:38.470 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:43:38.470 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:38.470 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 1 librados: starting msgr at 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 1 librados: starting objecter 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 1 Processor -- start 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 1 -- start start 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 1 librados: setting wanted keys 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 1 librados: calling monclient init 2024-03-25T19:43:38.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.470+0000 7fe8731f6640 10 monclient: init 2024-03-25T19:43:38.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:38.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:38.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe86c0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:38.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe86c13f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:38.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe86c1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:38.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c1d7c00 con 0x7fe86c1430c0 2024-03-25T19:43:38.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c0b2430 con 0x7fe86c0a98f0 2024-03-25T19:43:38.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c0b1970 con 0x7fe86c13f710 2024-03-25T19:43:38.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:38.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.471+0000 7fe8731f6640 10 monclient(hunting): authenticate will time out at 4389.612560s 2024-03-25T19:43:38.481 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.481+0000 7fe871793640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe86c1430c0 0x7fe86c1d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:47778/0 (socket says 172.21.15.110:47778) 2024-03-25T19:43:38.481 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.481+0000 7fe871793640 1 -- 172.21.15.110:0/2251403029 learned_addr learned my addr 172.21.15.110:0/2251403029 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:38.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.482+0000 7fe8697fa640 10 client.?.objecter ms_handle_connect 0x7fe86c1430c0 2024-03-25T19:43:38.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.482+0000 7fe8697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.485+0000 7fe8697fa640 10 client.?.objecter ms_handle_connect 0x7fe86c0a98f0 2024-03-25T19:43:38.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.485+0000 7fe8697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.498+0000 7fe8697fa640 10 client.?.objecter ms_handle_connect 0x7fe86c13f710 2024-03-25T19:43:38.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.498+0000 7fe8697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.508+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101297177 0 0) 0x7fe86c1d7c00 con 0x7fe86c1430c0 2024-03-25T19:43:38.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.508+0000 7fe8697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.508+0000 7fe8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.508+0000 7fe8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.508+0000 7fe8697fa640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:43:38.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.508+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe850004eb0 con 0x7fe86c1430c0 2024-03-25T19:43:38.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.569+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085465568 0 0) 0x7fe86c0b2430 con 0x7fe86c0a98f0 2024-03-25T19:43:38.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.569+0000 7fe8697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.569+0000 7fe8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.569+0000 7fe8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.570 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.569+0000 7fe8697fa640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:43:38.570 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.569+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe86c1d7c00 con 0x7fe86c0a98f0 2024-03-25T19:43:38.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.577+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951432256 0 0) 0x7fe86c0b1970 con 0x7fe86c13f710 2024-03-25T19:43:38.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.577+0000 7fe8697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.577+0000 7fe8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.577+0000 7fe8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.577+0000 7fe8697fa640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:43:38.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.577+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe86c0b2430 con 0x7fe86c13f710 2024-03-25T19:43:38.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.581+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2178522752 0 0) 0x7fe850004eb0 con 0x7fe86c1430c0 2024-03-25T19:43:38.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.581+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe86c0b1970 con 0x7fe86c1430c0 2024-03-25T19:43:38.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.581+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe864003970 con 0x7fe86c1430c0 2024-03-25T19:43:38.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.581+0000 7fe8697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.581+0000 7fe8697fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.581+0000 7fe8697fa640 10 monclient(hunting): dump: 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:38.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.581+0000 7fe8697fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.587+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2041531602 0 0) 0x7fe86c1d7c00 con 0x7fe86c0a98f0 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.587+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe850004eb0 con 0x7fe86c0a98f0 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.587+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe8600018a0 con 0x7fe86c0a98f0 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.587+0000 7fe8697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.587+0000 7fe8697fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.587+0000 7fe8697fa640 10 monclient(hunting): dump: 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:38.587 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:38.588 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:38.588 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.588 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:38.588 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:38.588 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:38.588 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:38.588 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3398744206 0 0) 0x7fe86c0b1970 con 0x7fe86c1430c0 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 >> v1:172.21.15.110:6790/0 conn(0x7fe86c13f710 legacy=0x7fe86c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 >> v1:172.21.15.178:6789/0 conn(0x7fe86c0a98f0 legacy=0x7fe86c0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 1 monclient: found mon.a 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c1d5000 con 0x7fe86c1430c0 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 5 monclient: authenticate success, global_id 4365 2024-03-25T19:43:38.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 10 monclient: _renew_subs 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86c1d7600 con 0x7fe86c1430c0 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 10 monclient: _renew_subs 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86c14e120 con 0x7fe86c1430c0 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.588+0000 7fe8731f6640 1 librados: init done 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.590+0000 7fe8731f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.590+0000 7fe8731f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.590+0000 7fe8731f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:38.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.590+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86c1d5250 con 0x7fe86c1430c0 2024-03-25T19:43:38.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.897+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8640015d0 con 0x7fe86c1430c0 2024-03-25T19:43:38.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.897+0000 7fe8697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:38.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.897+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe864003f20 con 0x7fe86c1430c0 2024-03-25T19:43:38.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.897+0000 7fe8697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.897+0000 7fe8697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:38.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.897+0000 7fe8697fa640 10 monclient: dump: 2024-03-25T19:43:38.897 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:38.897 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:38.898 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.897+0000 7fe86affd640 4 set_mon_vals no callback set 2024-03-25T19:43:38.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.898+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fe864004080 con 0x7fe86c1430c0 2024-03-25T19:43:38.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.898+0000 7fe870f92640 10 monclient: get_auth_request con 0x7fe85006df70 auth_method 0 2024-03-25T19:43:38.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.899+0000 7fe8697fa640 10 client.4365.objecter ms_handle_connect 0x7fe85006df70 2024-03-25T19:43:38.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.903+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fe864054d10 con 0x7fe86c1430c0 2024-03-25T19:43:38.903 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.903+0000 7fe8697fa640 10 client.4365.objecter ms_dispatch 0x7fe86c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:38.903 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.903+0000 7fe8697fa640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:38.903 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.903+0000 7fe8697fa640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:38.903 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.903+0000 7fe8697fa640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T19:43:38.903 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.903+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe864050650 con 0x7fe86c1430c0 2024-03-25T19:43:38.903 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.903+0000 7fe8697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.903 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:38.903+0000 7fe8697fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:39.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.152+0000 7fe8731f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:39.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.152+0000 7fe8731f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:39.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.152+0000 7fe8731f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:39.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.152+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe86c074cc0 con 0x7fe86c1430c0 2024-03-25T19:43:39.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.183+0000 7fe8697fa640 1 -- 172.21.15.110:0/2251403029 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fe864052750 con 0x7fe86c1430c0 2024-03-25T19:43:39.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.183+0000 7fe8697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:39.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.183+0000 7fe8697fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:39.183 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:43:39.183 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:39.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 10 librados: watch_flush enter 2024-03-25T19:43:39.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 10 librados: watch_flush exit 2024-03-25T19:43:39.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 10 client.4365.objecter successfully canceled tick 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 >> v1:172.21.15.178:6800/1132739058 conn(0x7fe85006df70 legacy=0x7fe850070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 10 monclient: shutdown 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 >> v1:172.21.15.110:6789/0 conn(0x7fe86c1430c0 legacy=0x7fe86c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 shutdown_connections 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 >> 172.21.15.110:0/2251403029 conn(0x7fe86c0b7ea0 msgr2=0x7fe86c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 shutdown_connections 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 1 -- 172.21.15.110:0/2251403029 wait complete. 2024-03-25T19:43:39.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:39.185+0000 7fe8731f6640 1 librados: shutdown 2024-03-25T19:43:39.197 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:46.004 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:46.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.120+0000 7fb823f7c640 10 monclient: get_monmap_and_config 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.120+0000 7fb823f7c640 10 monclient: build_initial_monmap 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.120+0000 7fb823f7c640 10 monclient: monmap: 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:43:46.122107+0000 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:43:46.122107+0000 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:43:46.121 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:46.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.120+0000 7fb823f7c640 1 Processor -- start 2024-03-25T19:43:46.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 1 -- start start 2024-03-25T19:43:46.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 10 monclient: init 2024-03-25T19:43:46.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:46.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 10 monclient(hunting): picked mon.noname-c con 0x7fb81c0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 10 monclient(hunting): picked mon.noname-b con 0x7fb81c13f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 10 monclient(hunting): picked mon.noname-a con 0x7fb81c1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c142520 con 0x7fb81c1430c0 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c0b28a0 con 0x7fb81c0a98f0 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c0b2a30 con 0x7fb81c13f710 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.121+0000 7fb823f7c640 10 monclient(hunting): authenticate will time out at 4397.263805s 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.128+0000 7fb822519640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb81c1430c0 0x7fb81c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:43448/0 (socket says 172.21.15.110:43448) 2024-03-25T19:43:46.129 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.128+0000 7fb822519640 1 -- 172.21.15.110:0/850856973 learned_addr learned my addr 172.21.15.110:0/850856973 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:46.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.142+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700036545 0 0) 0x7fb81c0b2a30 con 0x7fb81c13f710 2024-03-25T19:43:46.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.142+0000 7fb820d16640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.142+0000 7fb820d16640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.142+0000 7fb820d16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.142+0000 7fb820d16640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:43:46.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.142+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb808003520 con 0x7fb81c13f710 2024-03-25T19:43:46.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.144+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720049397 0 0) 0x7fb81c142520 con 0x7fb81c1430c0 2024-03-25T19:43:46.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.144+0000 7fb820d16640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.144+0000 7fb820d16640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.144+0000 7fb820d16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.144+0000 7fb820d16640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:43:46.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.144+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb81c0b2a30 con 0x7fb81c1430c0 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.153+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002465936 0 0) 0x7fb81c0b28a0 con 0x7fb81c0a98f0 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.153+0000 7fb820d16640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.153+0000 7fb820d16640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.153+0000 7fb820d16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.153+0000 7fb820d16640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb81c142520 con 0x7fb81c0a98f0 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1625141628 0 0) 0x7fb81c0b2a30 con 0x7fb81c1430c0 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 >> v1:172.21.15.110:6790/0 conn(0x7fb81c13f710 legacy=0x7fb81c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 >> v1:172.21.15.178:6789/0 conn(0x7fb81c0a98f0 legacy=0x7fb81c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 10 monclient: _finish_hunting 0 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 1 monclient: found mon.noname-a 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c0b44c0 con 0x7fb81c1430c0 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 10 monclient: _finish_auth 0 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 10 monclient: _check_auth_tickets 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb8140019f0 con 0x7fb81c1430c0 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb823f7c640 5 monclient: authenticate success, global_id 4368 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb823f7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb820d16640 10 monclient: dump: 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:46.156 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.157 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.157 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.157 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:46.157 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:46.157 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:46.157 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:46.157 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:46.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.154+0000 7fb823f7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:46.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.163+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb814001b50 con 0x7fb81c1430c0 2024-03-25T19:43:46.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.163+0000 7fb820d16640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:46.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.163+0000 7fb820d16640 1 -- 172.21.15.110:0/850856973 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb814001f60 con 0x7fb81c1430c0 2024-03-25T19:43:46.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.163+0000 7fb823f7c640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:46.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.163+0000 7fb823f7c640 4 set_mon_vals no callback set 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.163+0000 7fb820d16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.163+0000 7fb820d16640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.163+0000 7fb820d16640 10 monclient: dump: 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:46.164 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:46.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.164+0000 7fb823f7c640 10 monclient: shutdown 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.164+0000 7fb823f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.164+0000 7fb823f7c640 1 -- 172.21.15.110:0/850856973 >> v1:172.21.15.110:6789/0 conn(0x7fb81c1430c0 legacy=0x7fb81c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.164+0000 7fb823f7c640 1 -- 172.21.15.110:0/850856973 shutdown_connections 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.164+0000 7fb823f7c640 1 -- 172.21.15.110:0/850856973 >> 172.21.15.110:0/850856973 conn(0x7fb81c0b7ea0 msgr2=0x7fb81c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.164+0000 7fb823f7c640 1 -- 172.21.15.110:0/850856973 shutdown_connections 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.164+0000 7fb823f7c640 1 -- 172.21.15.110:0/850856973 wait complete. 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.165+0000 7fb823f7c640 10 monclient: build_initial_monmap 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.165+0000 7fb823f7c640 10 monclient: monmap: 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:46.166 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:43:46.167 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:46.167 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:43:46.167 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:43:46.167 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.165+0000 7fb823f7c640 1 librados: starting msgr at 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.165+0000 7fb823f7c640 1 librados: starting objecter 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.165+0000 7fb823f7c640 1 Processor -- start 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 1 -- start start 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 1 librados: setting wanted keys 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 1 librados: calling monclient init 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 10 monclient: init 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:46.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:46.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 10 monclient(hunting): picked mon.noname-a con 0x7fb81c0a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:46.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 10 monclient(hunting): picked mon.noname-c con 0x7fb81c13f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:46.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 10 monclient(hunting): picked mon.noname-b con 0x7fb81c1430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:46.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c138bd0 con 0x7fb81c0a98f0 2024-03-25T19:43:46.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c0aba30 con 0x7fb81c1430c0 2024-03-25T19:43:46.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c137740 con 0x7fb81c13f710 2024-03-25T19:43:46.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:46.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.166+0000 7fb823f7c640 10 monclient(hunting): authenticate will time out at 4397.308697s 2024-03-25T19:43:46.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.171+0000 7fb822519640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fb81c1430c0 0x7fb81c1d7780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:37258/0 (socket says 172.21.15.110:37258) 2024-03-25T19:43:46.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.171+0000 7fb822519640 1 -- 172.21.15.110:0/3108545036 learned_addr learned my addr 172.21.15.110:0/3108545036 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:46.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.172+0000 7fb8127fc640 10 client.?.objecter ms_handle_connect 0x7fb81c1430c0 2024-03-25T19:43:46.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.172+0000 7fb8127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:46.181 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.181+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206737133 0 0) 0x7fb81c0aba30 con 0x7fb81c1430c0 2024-03-25T19:43:46.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.181+0000 7fb8127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.181+0000 7fb8127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.181+0000 7fb8127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.181+0000 7fb8127fc640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:43:46.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.181+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb800004eb0 con 0x7fb81c1430c0 2024-03-25T19:43:46.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.182+0000 7fb8127fc640 10 client.?.objecter ms_handle_connect 0x7fb81c13f710 2024-03-25T19:43:46.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.182+0000 7fb8127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:46.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.191+0000 7fb8127fc640 10 client.?.objecter ms_handle_connect 0x7fb81c0a98f0 2024-03-25T19:43:46.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.191+0000 7fb8127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:46.198 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.198+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223507888 0 0) 0x7fb81c137740 con 0x7fb81c13f710 2024-03-25T19:43:46.198 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.198+0000 7fb8127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.198 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.198+0000 7fb8127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.198 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.198+0000 7fb8127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.198 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.198+0000 7fb8127fc640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:43:46.198 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.198+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb81c0aba30 con 0x7fb81c13f710 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.202+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 798217215 0 0) 0x7fb800004eb0 con 0x7fb81c1430c0 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.202+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb81c137740 con 0x7fb81c1430c0 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.202+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb814001f60 con 0x7fb81c1430c0 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.202+0000 7fb8127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.202+0000 7fb8127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.202+0000 7fb8127fc640 10 monclient(hunting): dump: 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:46.203 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:46.204 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:46.204 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:46.204 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.202+0000 7fb8127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:46.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.210+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065309944 0 0) 0x7fb81c138bd0 con 0x7fb81c0a98f0 2024-03-25T19:43:46.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.210+0000 7fb8127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.210+0000 7fb8127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.210+0000 7fb8127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.210+0000 7fb8127fc640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:43:46.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.210+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb800004eb0 con 0x7fb81c0a98f0 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.220+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3684357999 0 0) 0x7fb81c0aba30 con 0x7fb81c13f710 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.220+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb81c138bd0 con 0x7fb81c13f710 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.220+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb80c001950 con 0x7fb81c13f710 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.220+0000 7fb8127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.220+0000 7fb8127fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.220+0000 7fb8127fc640 10 monclient(hunting): dump: 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.221 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.222 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.222 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:46.222 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:46.222 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:46.222 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:46.222 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3019775146 0 0) 0x7fb81c137740 con 0x7fb81c1430c0 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 >> v1:172.21.15.110:6790/0 conn(0x7fb81c13f710 legacy=0x7fb81c134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 >> v1:172.21.15.110:6789/0 conn(0x7fb81c0a98f0 legacy=0x7fb81c1384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 1 monclient: found mon.b 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c134de0 con 0x7fb81c1430c0 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb8127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:46.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 5 monclient: authenticate success, global_id 4378 2024-03-25T19:43:46.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 10 monclient: _renew_subs 2024-03-25T19:43:46.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:46.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c0b1620 con 0x7fb81c1430c0 2024-03-25T19:43:46.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:46.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 10 monclient: _renew_subs 2024-03-25T19:43:46.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:46.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c134900 con 0x7fb81c1430c0 2024-03-25T19:43:46.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.224+0000 7fb823f7c640 1 librados: init done 2024-03-25T19:43:46.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.225+0000 7fb823f7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:46.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.225+0000 7fb823f7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:46.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.225+0000 7fb823f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:46.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.225+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c1d75d0 con 0x7fb81c1430c0 2024-03-25T19:43:46.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.516+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb814004e10 con 0x7fb81c1430c0 2024-03-25T19:43:46.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.516+0000 7fb8127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:46.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.516+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb814005120 con 0x7fb81c1430c0 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.516+0000 7fb8127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.517+0000 7fb8127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.517+0000 7fb8127fc640 10 monclient: dump: 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:46.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.516+0000 7fb813fff640 4 set_mon_vals no callback set 2024-03-25T19:43:46.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.518+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fb81401b0f0 con 0x7fb81c1430c0 2024-03-25T19:43:46.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.518+0000 7fb821d18640 10 monclient: get_auth_request con 0x7fb80006df70 auth_method 0 2024-03-25T19:43:46.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.519+0000 7fb8127fc640 10 client.4378.objecter ms_handle_connect 0x7fb80006df70 2024-03-25T19:43:46.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.524+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fb814054c20 con 0x7fb81c1430c0 2024-03-25T19:43:46.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.524+0000 7fb8127fc640 10 client.4378.objecter ms_dispatch 0x7fb81c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:46.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.524+0000 7fb8127fc640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:46.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.524+0000 7fb8127fc640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:46.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.524+0000 7fb8127fc640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T19:43:46.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.524+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb814050730 con 0x7fb81c1430c0 2024-03-25T19:43:46.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.524+0000 7fb8127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:46.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.524+0000 7fb8127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:46.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.752+0000 7fb823f7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.752+0000 7fb823f7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.752+0000 7fb823f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:43:46.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.752+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb81c0627b0 con 0x7fb81c1430c0 2024-03-25T19:43:46.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.782+0000 7fb8127fc640 1 -- 172.21.15.110:0/3108545036 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fb814052660 con 0x7fb81c1430c0 2024-03-25T19:43:46.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.782+0000 7fb8127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.783 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.782+0000 7fb8127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:46.783 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:43:46.783 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 10 librados: watch_flush enter 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 10 librados: watch_flush exit 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 10 client.4378.objecter successfully canceled tick 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 >> v1:172.21.15.178:6800/1132739058 conn(0x7fb80006df70 legacy=0x7fb800070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 10 monclient: shutdown 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 >> v1:172.21.15.178:6789/0 conn(0x7fb81c1430c0 legacy=0x7fb81c1d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 shutdown_connections 2024-03-25T19:43:46.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 >> 172.21.15.110:0/3108545036 conn(0x7fb81c0b7ea0 msgr2=0x7fb81c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:46.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 shutdown_connections 2024-03-25T19:43:46.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 1 -- 172.21.15.110:0/3108545036 wait complete. 2024-03-25T19:43:46.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:46.785+0000 7fb823f7c640 1 librados: shutdown 2024-03-25T19:43:46.797 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:53.604 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:53.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.700+0000 7faeb5aba640 10 monclient: get_monmap_and_config 2024-03-25T19:43:53.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.700+0000 7faeb5aba640 10 monclient: build_initial_monmap 2024-03-25T19:43:53.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.700+0000 7faeb5aba640 10 monclient: monmap: 2024-03-25T19:43:53.700 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:43:53.701717+0000 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:43:53.701717+0000 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:43:53.701 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:53.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.701+0000 7faeb5aba640 1 Processor -- start 2024-03-25T19:43:53.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.701+0000 7faeb5aba640 1 -- start start 2024-03-25T19:43:53.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.701+0000 7faeb5aba640 10 monclient: init 2024-03-25T19:43:53.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 10 monclient(hunting): picked mon.noname-b con 0x7faeb00a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 10 monclient(hunting): picked mon.noname-a con 0x7faeb013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 10 monclient(hunting): picked mon.noname-c con 0x7faeb01430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb0142520 con 0x7faeb013f710 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb00b28a0 con 0x7faeb01430c0 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb00b2a30 con 0x7faeb00a98f0 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:53.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.702+0000 7faeb5aba640 10 monclient(hunting): authenticate will time out at 4404.843510s 2024-03-25T19:43:53.718 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.719+0000 7faea7fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7faeb013f710 0x7faeb01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59630/0 (socket says 172.21.15.110:59630) 2024-03-25T19:43:53.718 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.719+0000 7faea7fff640 1 -- 172.21.15.110:0/4033875803 learned_addr learned my addr 172.21.15.110:0/4033875803 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.730+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359119410 0 0) 0x7faeb00b28a0 con 0x7faeb01430c0 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.730+0000 7faeaeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.730+0000 7faeaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.730+0000 7faeaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.730+0000 7faeaeffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.730+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae9c003520 con 0x7faeb01430c0 2024-03-25T19:43:53.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.740+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857188634 0 0) 0x7faeb00b2a30 con 0x7faeb00a98f0 2024-03-25T19:43:53.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.741+0000 7faeaeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.741+0000 7faeaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.741+0000 7faeaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.741+0000 7faeaeffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:43:53.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.741+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb00b28a0 con 0x7faeb00a98f0 2024-03-25T19:43:53.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.742+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779556366 0 0) 0x7faeb0142520 con 0x7faeb013f710 2024-03-25T19:43:53.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.742+0000 7faeaeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.742+0000 7faeaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.742+0000 7faeaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.742+0000 7faeaeffd640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:43:53.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.742+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb00b2a30 con 0x7faeb013f710 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534076807 0 0) 0x7fae9c003520 con 0x7faeb01430c0 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 >> v1:172.21.15.110:6790/0 conn(0x7faeb00a98f0 legacy=0x7faeb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 >> v1:172.21.15.110:6789/0 conn(0x7faeb013f710 legacy=0x7faeb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 1 monclient: found mon.noname-c 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb00b44c0 con 0x7faeb01430c0 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:53.745 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faea8001a10 con 0x7faeb01430c0 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeaeffd640 10 monclient: dump: 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:53.746 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:53.747 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:53.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeb5aba640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:43:53.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.745+0000 7faeb5aba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.767+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faea8001b70 con 0x7faeb01430c0 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.767+0000 7faeaeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.767+0000 7faeaeffd640 1 -- 172.21.15.110:0/4033875803 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faea8001f80 con 0x7faeb01430c0 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.767+0000 7faeb5aba640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.767+0000 7faeb5aba640 4 set_mon_vals no callback set 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.767+0000 7faeaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.767+0000 7faeaeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.767+0000 7faeaeffd640 10 monclient: dump: 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.767 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.768 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.768 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:53.768 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:53.768 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:53.768 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:53.768 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:53.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.768+0000 7faeb5aba640 10 monclient: shutdown 2024-03-25T19:43:53.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.768+0000 7faeb5aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:53.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.768+0000 7faeb5aba640 1 -- 172.21.15.110:0/4033875803 >> v1:172.21.15.178:6789/0 conn(0x7faeb01430c0 legacy=0x7faeb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.768+0000 7faeb5aba640 1 -- 172.21.15.110:0/4033875803 shutdown_connections 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.768+0000 7faeb5aba640 1 -- 172.21.15.110:0/4033875803 >> 172.21.15.110:0/4033875803 conn(0x7faeb00b7ea0 msgr2=0x7faeb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.768+0000 7faeb5aba640 1 -- 172.21.15.110:0/4033875803 shutdown_connections 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.768+0000 7faeb5aba640 1 -- 172.21.15.110:0/4033875803 wait complete. 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.769+0000 7faeb5aba640 10 monclient: build_initial_monmap 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.769+0000 7faeb5aba640 10 monclient: monmap: 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:43:53.770 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:53.771 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.770+0000 7faeb5aba640 1 librados: starting msgr at 2024-03-25T19:43:53.771 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.770+0000 7faeb5aba640 1 librados: starting objecter 2024-03-25T19:43:53.771 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.770+0000 7faeb5aba640 1 Processor -- start 2024-03-25T19:43:53.771 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.770+0000 7faeb5aba640 1 -- start start 2024-03-25T19:43:53.771 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.770+0000 7faeb5aba640 1 librados: setting wanted keys 2024-03-25T19:43:53.771 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.770+0000 7faeb5aba640 1 librados: calling monclient init 2024-03-25T19:43:53.771 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.770+0000 7faeb5aba640 10 monclient: init 2024-03-25T19:43:53.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:53.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:53.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 10 monclient(hunting): picked mon.noname-b con 0x7faeb00a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:43:53.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 10 monclient(hunting): picked mon.noname-c con 0x7faeb013f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:43:53.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 10 monclient(hunting): picked mon.noname-a con 0x7faeb01430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:43:53.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb0135230 con 0x7faeb01430c0 2024-03-25T19:43:53.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb0135da0 con 0x7faeb00a98f0 2024-03-25T19:43:53.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb0135f30 con 0x7faeb013f710 2024-03-25T19:43:53.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:53.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.771+0000 7faeb5aba640 10 monclient(hunting): authenticate will time out at 4404.912158s 2024-03-25T19:43:53.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.779+0000 7faea7fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7faeb013f710 0x7faeb01dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:37876/0 (socket says 172.21.15.110:37876) 2024-03-25T19:43:53.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.779+0000 7faea7fff640 1 -- 172.21.15.110:0/664329955 learned_addr learned my addr 172.21.15.110:0/664329955 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:43:53.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.780+0000 7faeacff9640 10 client.?.objecter ms_handle_connect 0x7faeb013f710 2024-03-25T19:43:53.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.780+0000 7faeacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.786 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.787+0000 7faeacff9640 10 client.?.objecter ms_handle_connect 0x7faeb00a98f0 2024-03-25T19:43:53.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.787+0000 7faeacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.787+0000 7faeacff9640 10 client.?.objecter ms_handle_connect 0x7faeb01430c0 2024-03-25T19:43:53.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.787+0000 7faeacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.795+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828858551 0 0) 0x7faeb0135da0 con 0x7faeb00a98f0 2024-03-25T19:43:53.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.795+0000 7faeacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.795+0000 7faeacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.795+0000 7faeacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.795+0000 7faeacff9640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:43:53.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.795+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae90004eb0 con 0x7faeb00a98f0 2024-03-25T19:43:53.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.801+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633655660 0 0) 0x7faeb0135f30 con 0x7faeb013f710 2024-03-25T19:43:53.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.801+0000 7faeacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.801+0000 7faeacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.801+0000 7faeacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.801+0000 7faeacff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:43:53.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.801+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb0135da0 con 0x7faeb013f710 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.806+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3848417848 0 0) 0x7fae90004eb0 con 0x7faeb00a98f0 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.806+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faeb0135f30 con 0x7faeb00a98f0 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.806+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faea0001970 con 0x7faeb00a98f0 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.806+0000 7faeacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.806+0000 7faeacff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.806+0000 7faeacff9640 10 monclient(hunting): dump: 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:53.806 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:53.807 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:53.807 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:53.807 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:53.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.806+0000 7faeacff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:53.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.809+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062870792 0 0) 0x7faeb0135230 con 0x7faeb01430c0 2024-03-25T19:43:53.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.809+0000 7faeacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.809+0000 7faeacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.809+0000 7faeacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.809+0000 7faeacff9640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:43:53.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.809+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae90004eb0 con 0x7faeb01430c0 2024-03-25T19:43:53.818 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.818+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4280077823 0 0) 0x7fae90004eb0 con 0x7faeb01430c0 2024-03-25T19:43:53.818 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.818+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faeb0135230 con 0x7faeb01430c0 2024-03-25T19:43:53.818 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.818+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faea8001e80 con 0x7faeb01430c0 2024-03-25T19:43:53.818 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.818+0000 7faeacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.818 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.818+0000 7faeacff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:53.818 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.818+0000 7faeacff9640 10 monclient(hunting): dump: 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:53.819 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.827+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 415966681 0 0) 0x7faeb0135da0 con 0x7faeb013f710 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.827+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae90004eb0 con 0x7faeb013f710 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.827+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fae980018f0 con 0x7faeb013f710 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.827+0000 7faeacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.827+0000 7faeacff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.827+0000 7faeacff9640 10 monclient(hunting): dump: 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:53.827 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.828 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:53.828 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.828 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:53.828 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:53.828 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:53.828 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:53.828 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.829+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 141713102 0 0) 0x7faeb0135230 con 0x7faeb01430c0 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 >> v1:172.21.15.110:6790/0 conn(0x7faeb013f710 legacy=0x7faeb01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 >> v1:172.21.15.178:6789/0 conn(0x7faeb00a98f0 legacy=0x7faeb01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 1 monclient: found mon.a 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb01360c0 con 0x7faeb01430c0 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:53.830 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 5 monclient: authenticate success, global_id 4377 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 10 monclient: _renew_subs 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb01d5000 con 0x7faeb01430c0 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 10 monclient: _renew_subs 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb01d41e0 con 0x7faeb01430c0 2024-03-25T19:43:53.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.830+0000 7faeb5aba640 1 librados: init done 2024-03-25T19:43:53.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.831+0000 7faeb5aba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.831+0000 7faeb5aba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.831+0000 7faeb5aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:53.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:53.831+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb0135090 con 0x7faeb01430c0 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.115+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faea8001560 con 0x7faeb01430c0 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.115+0000 7faeacff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.115+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faea8001fe0 con 0x7faeb01430c0 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.115+0000 7faeacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.115+0000 7faeacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.115+0000 7faeacff9640 10 monclient: dump: 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:43:54.116 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:43:54.117 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:43:54.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.115+0000 7faeae7fc640 4 set_mon_vals no callback set 2024-03-25T19:43:54.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.116+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7faea8019a90 con 0x7faeb01430c0 2024-03-25T19:43:54.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.117+0000 7faeaf7fe640 10 monclient: get_auth_request con 0x7fae9006e080 auth_method 0 2024-03-25T19:43:54.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.117+0000 7faeacff9640 10 client.4377.objecter ms_handle_connect 0x7fae9006e080 2024-03-25T19:43:54.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.122+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7faea8053670 con 0x7faeb01430c0 2024-03-25T19:43:54.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.122+0000 7faeacff9640 10 client.4377.objecter ms_dispatch 0x7faeb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:54.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.122+0000 7faeacff9640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:54.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.122+0000 7faeacff9640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:54.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.122+0000 7faeacff9640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-25T19:43:54.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.122+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faea80b08b0 con 0x7faeb01430c0 2024-03-25T19:43:54.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.122+0000 7faeacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:54.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.122+0000 7faeacff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:54.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.379+0000 7faeb5aba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:54.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.379+0000 7faeb5aba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:54.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.379+0000 7faeb5aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:43:54.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.379+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeb013b6e0 con 0x7faeb01430c0 2024-03-25T19:43:54.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.411+0000 7faeacff9640 1 -- 172.21.15.110:0/664329955 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7faea804efb0 con 0x7faeb01430c0 2024-03-25T19:43:54.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.411+0000 7faeacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:54.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.411+0000 7faeacff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:54.411 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:43:54.411 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.413+0000 7faeb5aba640 10 librados: watch_flush enter 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.413+0000 7faeb5aba640 10 librados: watch_flush exit 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.413+0000 7faeb5aba640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.413+0000 7faeb5aba640 10 client.4377.objecter successfully canceled tick 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 >> v1:172.21.15.178:6800/1132739058 conn(0x7fae9006e080 legacy=0x7fae90070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 10 monclient: shutdown 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 >> v1:172.21.15.110:6789/0 conn(0x7faeb01430c0 legacy=0x7faeb01db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 shutdown_connections 2024-03-25T19:43:54.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 >> 172.21.15.110:0/664329955 conn(0x7faeb00b7ea0 msgr2=0x7faeb014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:54.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 shutdown_connections 2024-03-25T19:43:54.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 1 -- 172.21.15.110:0/664329955 wait complete. 2024-03-25T19:43:54.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:43:54.414+0000 7faeb5aba640 1 librados: shutdown 2024-03-25T19:43:54.425 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:57.432 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:04.716 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:57.443 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:04.727 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:57.474 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:04.758 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:57.499 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:04.754 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:57.515 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:04.770 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:57.521 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:04.805 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:57.532 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:04.816 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:57.550 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:04.805 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:57.564 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:04.848 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:01.231 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.344+0000 7f29c71dd640 10 monclient: get_monmap_and_config 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.344+0000 7f29c71dd640 10 monclient: build_initial_monmap 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.344+0000 7f29c71dd640 10 monclient: monmap: 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:44:01.345300+0000 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:44:01.345300+0000 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:01.344 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:01.345 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:01.345 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:44:01.345 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:44:01.345 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:01.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 1 Processor -- start 2024-03-25T19:44:01.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 1 -- start start 2024-03-25T19:44:01.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 10 monclient: init 2024-03-25T19:44:01.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:01.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:01.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 10 monclient(hunting): picked mon.noname-a con 0x7f29c00a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:01.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 10 monclient(hunting): picked mon.noname-b con 0x7f29c013f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:01.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 10 monclient(hunting): picked mon.noname-c con 0x7f29c01430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:01.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c0142520 con 0x7f29c00a98f0 2024-03-25T19:44:01.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c00b28a0 con 0x7f29c01430c0 2024-03-25T19:44:01.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c00b2a30 con 0x7f29c013f710 2024-03-25T19:44:01.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:01.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.345+0000 7f29c71dd640 10 monclient(hunting): authenticate will time out at 4412.486640s 2024-03-25T19:44:01.357 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.357+0000 7f29c577a640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f29c01430c0 0x7f29c014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:41866/0 (socket says 172.21.15.110:41866) 2024-03-25T19:44:01.357 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.357+0000 7f29c577a640 1 -- 172.21.15.110:0/3766860924 learned_addr learned my addr 172.21.15.110:0/3766860924 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.370+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486690331 0 0) 0x7f29c00b28a0 con 0x7f29c01430c0 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.370+0000 7f29bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.370+0000 7f29bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.370+0000 7f29bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.370+0000 7f29bf7fe640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:44:01.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.370+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b0003520 con 0x7f29c01430c0 2024-03-25T19:44:01.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.379+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838738572 0 0) 0x7f29c00b2a30 con 0x7f29c013f710 2024-03-25T19:44:01.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.379+0000 7f29bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.379+0000 7f29bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.379+0000 7f29bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.379+0000 7f29bf7fe640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:44:01.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.379+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c00b28a0 con 0x7f29c013f710 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738536200 0 0) 0x7f29b0003520 con 0x7f29c01430c0 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 >> v1:172.21.15.110:6790/0 conn(0x7f29c013f710 legacy=0x7f29c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 >> v1:172.21.15.110:6789/0 conn(0x7f29c00a98f0 legacy=0x7f29c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 1 monclient: found mon.noname-c 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c00b44c0 con 0x7f29c01430c0 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f29b8001a10 con 0x7f29c01430c0 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.384+0000 7f29bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.385+0000 7f29bf7fe640 10 monclient: dump: 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.385+0000 7f29c71dd640 5 monclient: authenticate success, global_id 4387 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.385+0000 7f29c71dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.393+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29b8001b70 con 0x7f29c01430c0 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.393+0000 7f29bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.393+0000 7f29bf7fe640 1 -- 172.21.15.110:0/3766860924 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f29b8001f80 con 0x7f29c01430c0 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.393+0000 7f29c71dd640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.393+0000 7f29c71dd640 4 set_mon_vals no callback set 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.393+0000 7f29bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.393+0000 7f29bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.394+0000 7f29bf7fe640 10 monclient: dump: 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.394+0000 7f29c71dd640 10 monclient: shutdown 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.394+0000 7f29c71dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.394+0000 7f29c71dd640 1 -- 172.21.15.110:0/3766860924 >> v1:172.21.15.178:6789/0 conn(0x7f29c01430c0 legacy=0x7f29c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.394+0000 7f29c71dd640 1 -- 172.21.15.110:0/3766860924 shutdown_connections 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.394+0000 7f29c71dd640 1 -- 172.21.15.110:0/3766860924 >> 172.21.15.110:0/3766860924 conn(0x7f29c00b7ea0 msgr2=0x7f29c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.394+0000 7f29c71dd640 1 -- 172.21.15.110:0/3766860924 shutdown_connections 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.394+0000 7f29c71dd640 1 -- 172.21.15.110:0/3766860924 wait complete. 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 10 monclient: build_initial_monmap 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 10 monclient: monmap: 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 1 librados: starting msgr at 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 1 librados: starting objecter 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 1 Processor -- start 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 1 -- start start 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 1 librados: setting wanted keys 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 1 librados: calling monclient init 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.395+0000 7f29c71dd640 10 monclient: init 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:01.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 10 monclient(hunting): picked mon.noname-a con 0x7f29c00a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:01.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 10 monclient(hunting): picked mon.noname-b con 0x7f29c013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:01.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 10 monclient(hunting): picked mon.noname-c con 0x7f29c01430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:01.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c01d7500 con 0x7f29c00a98f0 2024-03-25T19:44:01.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c01394a0 con 0x7f29c013f710 2024-03-25T19:44:01.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c0138b50 con 0x7f29c01430c0 2024-03-25T19:44:01.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:01.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.396+0000 7f29c71dd640 10 monclient(hunting): authenticate will time out at 4412.537513s 2024-03-25T19:44:01.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.407+0000 7f29bffff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f29c013f710 0x7f29c01d7c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:41910/0 (socket says 172.21.15.110:41910) 2024-03-25T19:44:01.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.407+0000 7f29bffff640 1 -- 172.21.15.110:0/4028861010 learned_addr learned my addr 172.21.15.110:0/4028861010 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:01.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.408+0000 7f29bd7fa640 10 client.?.objecter ms_handle_connect 0x7f29c013f710 2024-03-25T19:44:01.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.408+0000 7f29bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.415+0000 7f29bd7fa640 10 client.?.objecter ms_handle_connect 0x7f29c01430c0 2024-03-25T19:44:01.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.415+0000 7f29bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.421+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089621067 0 0) 0x7f29c01394a0 con 0x7f29c013f710 2024-03-25T19:44:01.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.421+0000 7f29bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.421+0000 7f29bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.421+0000 7f29bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.421+0000 7f29bd7fa640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:44:01.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.421+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a4004eb0 con 0x7f29c013f710 2024-03-25T19:44:01.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.424+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924001924 0 0) 0x7f29c0138b50 con 0x7f29c01430c0 2024-03-25T19:44:01.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.424+0000 7f29bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.424+0000 7f29bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.424+0000 7f29bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.424+0000 7f29bd7fa640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:44:01.426 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.424+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c01394a0 con 0x7f29c01430c0 2024-03-25T19:44:01.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.427+0000 7f29bd7fa640 10 client.?.objecter ms_handle_connect 0x7f29c00a98f0 2024-03-25T19:44:01.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.427+0000 7f29bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.446+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1778231902 0 0) 0x7f29c01394a0 con 0x7f29c01430c0 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.446+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29c0138b50 con 0x7f29c01430c0 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.446+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f29b8001e80 con 0x7f29c01430c0 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.446+0000 7f29bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.446+0000 7f29bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.446+0000 7f29bd7fa640 10 monclient(hunting): dump: 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:01.446 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.447 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.447 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.447 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:01.447 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:01.447 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:01.447 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:01.447 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:01.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.446+0000 7f29bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:01.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.448+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153939423 0 0) 0x7f29c01d7500 con 0x7f29c00a98f0 2024-03-25T19:44:01.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.448+0000 7f29bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.448+0000 7f29bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.448+0000 7f29bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.448+0000 7f29bd7fa640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:44:01.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.448+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c01394a0 con 0x7f29c00a98f0 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.451+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 619342938 0 0) 0x7f29a4004eb0 con 0x7f29c013f710 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.451+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29c01d7500 con 0x7f29c013f710 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.451+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f29ac001a10 con 0x7f29c013f710 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.451+0000 7f29bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.451+0000 7f29bd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.451+0000 7f29bd7fa640 10 monclient(hunting): dump: 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:01.451 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:01.452 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:01.452 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:01.452 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:01.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3953302320 0 0) 0x7f29c01d7500 con 0x7f29c013f710 2024-03-25T19:44:01.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 >> v1:172.21.15.110:6790/0 conn(0x7f29c01430c0 legacy=0x7f29c0134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 >> v1:172.21.15.110:6789/0 conn(0x7f29c00a98f0 legacy=0x7f29c0138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:01.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 1 monclient: found mon.b 2024-03-25T19:44:01.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:01.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:01.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c01d8380 con 0x7f29c013f710 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 10 monclient: _renew_subs 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c0139710 con 0x7f29c013f710 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 10 monclient: _renew_subs 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c01d7750 con 0x7f29c013f710 2024-03-25T19:44:01.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.470+0000 7f29c71dd640 1 librados: init done 2024-03-25T19:44:01.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.471+0000 7f29c71dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.471+0000 7f29c71dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.471+0000 7f29c71dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:01.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.471+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c00627b0 con 0x7f29c013f710 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.767+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29ac001c50 con 0x7f29c013f710 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.767+0000 7f29bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.767+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f29ac003a60 con 0x7f29c013f710 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.767+0000 7f29bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.767+0000 7f29bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.767+0000 7f29bd7fa640 10 monclient: dump: 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:01.767 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.767+0000 7f29beffd640 4 set_mon_vals no callback set 2024-03-25T19:44:01.768 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.767+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f29ac003c80 con 0x7f29c013f710 2024-03-25T19:44:01.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.768+0000 7f29c4f79640 10 monclient: get_auth_request con 0x7f29a406df50 auth_method 0 2024-03-25T19:44:01.769 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.768+0000 7f29bd7fa640 10 client.4390.objecter ms_handle_connect 0x7f29a406df50 2024-03-25T19:44:01.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.772+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f29ac054880 con 0x7f29c013f710 2024-03-25T19:44:01.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.772+0000 7f29bd7fa640 10 client.4390.objecter ms_dispatch 0x7f29c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:01.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.772+0000 7f29bd7fa640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:01.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.772+0000 7f29bd7fa640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:01.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.772+0000 7f29bd7fa640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T19:44:01.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.772+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29ac050390 con 0x7f29c013f710 2024-03-25T19:44:01.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.772+0000 7f29bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:01.772+0000 7f29bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:02.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.014+0000 7f29c71dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:02.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.014+0000 7f29c71dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:02.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.014+0000 7f29c71dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:02.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.014+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29c0069f40 con 0x7f29c013f710 2024-03-25T19:44:02.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.045+0000 7f29bd7fa640 1 -- 172.21.15.110:0/4028861010 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f29ac0522c0 con 0x7f29c013f710 2024-03-25T19:44:02.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.045+0000 7f29bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:02.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.045+0000 7f29bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:02.045 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:44:02.045 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:02.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 10 librados: watch_flush enter 2024-03-25T19:44:02.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 10 librados: watch_flush exit 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 10 client.4390.objecter successfully canceled tick 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 >> v1:172.21.15.178:6800/1132739058 conn(0x7f29a406df50 legacy=0x7f29a4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 10 monclient: shutdown 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 >> v1:172.21.15.178:6789/0 conn(0x7f29c013f710 legacy=0x7f29c01d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 shutdown_connections 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.047+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 >> 172.21.15.110:0/4028861010 conn(0x7f29c00b7ea0 msgr2=0x7f29c013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.048+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 shutdown_connections 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.048+0000 7f29c71dd640 1 -- 172.21.15.110:0/4028861010 wait complete. 2024-03-25T19:44:02.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:02.048+0000 7f29c71dd640 1 librados: shutdown 2024-03-25T19:44:02.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:08.866 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:08.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.978+0000 7f1e73c58640 10 monclient: get_monmap_and_config 2024-03-25T19:44:08.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.978+0000 7f1e73c58640 10 monclient: build_initial_monmap 2024-03-25T19:44:08.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.978+0000 7f1e73c58640 10 monclient: monmap: 2024-03-25T19:44:08.978 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:08.978 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:44:08.979208+0000 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:44:08.979208+0000 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:08.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.979+0000 7f1e73c58640 1 Processor -- start 2024-03-25T19:44:08.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.979+0000 7f1e73c58640 1 -- start start 2024-03-25T19:44:08.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.979+0000 7f1e73c58640 10 monclient: init 2024-03-25T19:44:08.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.979+0000 7f1e73c58640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:08.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.979+0000 7f1e73c58640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:08.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.979+0000 7f1e73c58640 10 monclient(hunting): picked mon.noname-a con 0x7f1e6c0a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:08.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.980+0000 7f1e73c58640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6c13f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:08.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.980+0000 7f1e73c58640 10 monclient(hunting): picked mon.noname-b con 0x7f1e6c1430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:08.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.980+0000 7f1e73c58640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c142520 con 0x7f1e6c0a98f0 2024-03-25T19:44:08.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.980+0000 7f1e73c58640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c0b28a0 con 0x7f1e6c13f710 2024-03-25T19:44:08.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.980+0000 7f1e73c58640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c0b2a30 con 0x7f1e6c1430c0 2024-03-25T19:44:08.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.980+0000 7f1e73c58640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:08.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.980+0000 7f1e73c58640 10 monclient(hunting): authenticate will time out at 4420.120927s 2024-03-25T19:44:08.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.990+0000 7f1e711f3640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f1e6c13f710 0x7f1e6c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:46342/0 (socket says 172.21.15.110:46342) 2024-03-25T19:44:08.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.990+0000 7f1e711f3640 1 -- 172.21.15.110:0/3598872633 learned_addr learned my addr 172.21.15.110:0/3598872633 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:08.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.998+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196589366 0 0) 0x7f1e6c142520 con 0x7f1e6c0a98f0 2024-03-25T19:44:08.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.998+0000 7f1e709f2640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.998+0000 7f1e709f2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.998+0000 7f1e709f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.998+0000 7f1e709f2640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:44:08.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:08.998+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e58003520 con 0x7f1e6c0a98f0 2024-03-25T19:44:08.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.000+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94634455 0 0) 0x7f1e6c0b28a0 con 0x7f1e6c13f710 2024-03-25T19:44:08.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.000+0000 7f1e709f2640 10 monclient(hunting): _negotiate 2024-03-25T19:44:09.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.000+0000 7f1e709f2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:09.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.000+0000 7f1e709f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:09.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.000+0000 7f1e709f2640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:44:09.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.000+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e6c142520 con 0x7f1e6c13f710 2024-03-25T19:44:09.003 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2820377408 0 0) 0x7f1e58003520 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 >> v1:172.21.15.110:6790/0 conn(0x7f1e6c1430c0 legacy=0x7f1e6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 >> v1:172.21.15.178:6789/0 conn(0x7f1e6c13f710 legacy=0x7f1e6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 10 monclient: _finish_hunting 0 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 1 monclient: found mon.noname-a 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c0b44c0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 10 monclient: _finish_auth 0 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 10 monclient: _check_auth_tickets 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e709f2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e73c58640 5 monclient: authenticate success, global_id 4386 2024-03-25T19:44:09.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.004+0000 7f1e73c58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:09.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.006+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1e68001650 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.006+0000 7f1e709f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:09.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.006+0000 7f1e709f2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.006+0000 7f1e709f2640 10 monclient: dump: 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:09.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.006+0000 7f1e73c58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:09.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.007+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e680018a0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.007+0000 7f1e709f2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:09.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.007+0000 7f1e709f2640 1 -- 172.21.15.110:0/3598872633 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1e68001c30 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.007+0000 7f1e709f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:09.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.007+0000 7f1e709f2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:09.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.007+0000 7f1e709f2640 10 monclient: dump: 2024-03-25T19:44:09.007 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.007+0000 7f1e73c58640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:09.008 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.007+0000 7f1e73c58640 4 set_mon_vals no callback set 2024-03-25T19:44:09.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.008+0000 7f1e73c58640 10 monclient: shutdown 2024-03-25T19:44:09.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.008+0000 7f1e73c58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:09.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.008+0000 7f1e73c58640 1 -- 172.21.15.110:0/3598872633 >> v1:172.21.15.110:6789/0 conn(0x7f1e6c0a98f0 legacy=0x7f1e6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.008+0000 7f1e73c58640 1 -- 172.21.15.110:0/3598872633 shutdown_connections 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.008+0000 7f1e73c58640 1 -- 172.21.15.110:0/3598872633 >> 172.21.15.110:0/3598872633 conn(0x7f1e6c0b7ea0 msgr2=0x7f1e6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.008+0000 7f1e73c58640 1 -- 172.21.15.110:0/3598872633 shutdown_connections 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.008+0000 7f1e73c58640 1 -- 172.21.15.110:0/3598872633 wait complete. 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.009+0000 7f1e73c58640 10 monclient: build_initial_monmap 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.009+0000 7f1e73c58640 10 monclient: monmap: 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:09.011 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.010+0000 7f1e73c58640 1 librados: starting msgr at 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.010+0000 7f1e73c58640 1 librados: starting objecter 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.010+0000 7f1e73c58640 1 Processor -- start 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.010+0000 7f1e73c58640 1 -- start start 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.010+0000 7f1e73c58640 1 librados: setting wanted keys 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.010+0000 7f1e73c58640 1 librados: calling monclient init 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.010+0000 7f1e73c58640 10 monclient: init 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 10 monclient(hunting): picked mon.noname-b con 0x7f1e6c0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 10 monclient(hunting): picked mon.noname-a con 0x7f1e6c13f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6c1430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c1d43a0 con 0x7f1e6c13f710 2024-03-25T19:44:09.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c1352b0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c135440 con 0x7f1e6c1430c0 2024-03-25T19:44:09.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:09.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.011+0000 7f1e73c58640 10 monclient(hunting): authenticate will time out at 4420.152192s 2024-03-25T19:44:09.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.021+0000 7f1e719f4640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f1e6c0a98f0 0x7f1e6c1387e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:46354/0 (socket says 172.21.15.110:46354) 2024-03-25T19:44:09.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.021+0000 7f1e719f4640 1 -- 172.21.15.110:0/2378585786 learned_addr learned my addr 172.21.15.110:0/2378585786 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:09.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.022+0000 7f1e627fc640 10 client.?.objecter ms_handle_connect 0x7f1e6c0a98f0 2024-03-25T19:44:09.021 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.022+0000 7f1e627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:09.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.024+0000 7f1e627fc640 10 client.?.objecter ms_handle_connect 0x7f1e6c1430c0 2024-03-25T19:44:09.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.024+0000 7f1e627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:09.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.030+0000 7f1e627fc640 10 client.?.objecter ms_handle_connect 0x7f1e6c13f710 2024-03-25T19:44:09.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.030+0000 7f1e627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:09.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.030+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716081308 0 0) 0x7f1e6c1352b0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.030+0000 7f1e627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:09.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.030+0000 7f1e627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:09.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.030+0000 7f1e627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:09.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.030+0000 7f1e627fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:44:09.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.030+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e50004eb0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.044+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576115100 0 0) 0x7f1e6c135440 con 0x7f1e6c1430c0 2024-03-25T19:44:09.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.044+0000 7f1e627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:09.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.044+0000 7f1e627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:09.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.044+0000 7f1e627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:09.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.044+0000 7f1e627fc640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:44:09.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.044+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e6c1352b0 con 0x7f1e6c1430c0 2024-03-25T19:44:09.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.047+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3650212616 0 0) 0x7f1e50004eb0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.047+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e6c135440 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.047+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1e68001c70 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.047+0000 7f1e627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.047+0000 7f1e627fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.047+0000 7f1e627fc640 10 monclient(hunting): dump: 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:09.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.047+0000 7f1e627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:09.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 520527181 0 0) 0x7f1e6c135440 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 >> v1:172.21.15.110:6790/0 conn(0x7f1e6c1430c0 legacy=0x7f1e6c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 >> v1:172.21.15.110:6789/0 conn(0x7f1e6c13f710 legacy=0x7f1e6c1d48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:09.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 1 monclient: found mon.b 2024-03-25T19:44:09.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c138ef0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 10 monclient: _renew_subs 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6c1d4fd0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 10 monclient: _renew_subs 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:09.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6c138200 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.054+0000 7f1e73c58640 1 librados: init done 2024-03-25T19:44:09.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.055+0000 7f1e73c58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:09.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.055+0000 7f1e73c58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:09.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.055+0000 7f1e73c58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:09.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.055+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e6c1d41e0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.325+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e68001230 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.325+0000 7f1e627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:09.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.325+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1e68003bc0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.325+0000 7f1e627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.325+0000 7f1e627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.325+0000 7f1e627fc640 10 monclient: dump: 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:09.326 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.325+0000 7f1e63fff640 4 set_mon_vals no callback set 2024-03-25T19:44:09.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.326+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f1e68003de0 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.327+0000 7f1e711f3640 10 monclient: get_auth_request con 0x7f1e5006dd10 auth_method 0 2024-03-25T19:44:09.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.327+0000 7f1e627fc640 10 client.4396.objecter ms_handle_connect 0x7f1e5006dd10 2024-03-25T19:44:09.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.332+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f1e68054950 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.332 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.332+0000 7f1e627fc640 10 client.4396.objecter ms_dispatch 0x7f1e6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:09.332 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.332+0000 7f1e627fc640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:09.332 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.332+0000 7f1e627fc640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:09.332 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.332+0000 7f1e627fc640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T19:44:09.332 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.332+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e68050460 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.332 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.333+0000 7f1e627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:09.332 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.333+0000 7f1e627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:09.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.591+0000 7f1e73c58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.591+0000 7f1e73c58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.591+0000 7f1e73c58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:09.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.591+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e6c1d4530 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.613+0000 7f1e627fc640 1 -- 172.21.15.110:0/2378585786 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f1e68052390 con 0x7f1e6c0a98f0 2024-03-25T19:44:09.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.613+0000 7f1e627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.613+0000 7f1e627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:09.613 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:44:09.613 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:09.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 10 librados: watch_flush enter 2024-03-25T19:44:09.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 10 librados: watch_flush exit 2024-03-25T19:44:09.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T19:44:09.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 10 client.4396.objecter successfully canceled tick 2024-03-25T19:44:09.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 >> v1:172.21.15.178:6800/1132739058 conn(0x7f1e5006dd10 legacy=0x7f1e500701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 10 monclient: shutdown 2024-03-25T19:44:09.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:09.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 >> v1:172.21.15.178:6789/0 conn(0x7f1e6c0a98f0 legacy=0x7f1e6c1387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 shutdown_connections 2024-03-25T19:44:09.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.616+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 >> 172.21.15.110:0/2378585786 conn(0x7f1e6c0b7ea0 msgr2=0x7f1e6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:09.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.617+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 shutdown_connections 2024-03-25T19:44:09.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.617+0000 7f1e73c58640 1 -- 172.21.15.110:0/2378585786 wait complete. 2024-03-25T19:44:09.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:09.617+0000 7f1e73c58640 1 librados: shutdown 2024-03-25T19:44:09.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:16.435 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:16.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.552+0000 7fdc4ea06640 10 monclient: get_monmap_and_config 2024-03-25T19:44:16.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.552+0000 7fdc4ea06640 10 monclient: build_initial_monmap 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.552+0000 7fdc4ea06640 10 monclient: monmap: 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:44:16.553080+0000 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:44:16.553080+0000 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:44:16.552 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 1 Processor -- start 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 1 -- start start 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 10 monclient: init 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 10 monclient(hunting): picked mon.noname-b con 0x7fdc480a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 10 monclient(hunting): picked mon.noname-c con 0x7fdc4813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 10 monclient(hunting): picked mon.noname-a con 0x7fdc481430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48142520 con 0x7fdc481430c0 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.553+0000 7fdc4ea06640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc480b28a0 con 0x7fdc4813f710 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.554+0000 7fdc4ea06640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc480b2a30 con 0x7fdc480a98f0 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.554+0000 7fdc4ea06640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:16.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.554+0000 7fdc4ea06640 10 monclient(hunting): authenticate will time out at 4427.694889s 2024-03-25T19:44:16.556 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.556+0000 7fdc47fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fdc480a98f0 0x7fdc480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40318/0 (socket says 172.21.15.110:40318) 2024-03-25T19:44:16.556 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.556+0000 7fdc47fff640 1 -- 172.21.15.110:0/2064438574 learned_addr learned my addr 172.21.15.110:0/2064438574 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:16.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.561+0000 7fdc46ffd640 1 -- 172.21.15.110:0/2064438574 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559319131 0 0) 0x7fdc480b2a30 con 0x7fdc480a98f0 2024-03-25T19:44:16.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.561+0000 7fdc46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.561+0000 7fdc46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.561+0000 7fdc46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.561+0000 7fdc46ffd640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:44:16.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.561+0000 7fdc46ffd640 1 -- 172.21.15.110:0/2064438574 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc30003520 con 0x7fdc480a98f0 2024-03-25T19:44:16.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 1 -- 172.21.15.110:0/2064438574 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 74496543 0 0) 0x7fdc30003520 con 0x7fdc480a98f0 2024-03-25T19:44:16.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 1 -- 172.21.15.110:0/2064438574 >> v1:172.21.15.178:6789/0 conn(0x7fdc4813f710 legacy=0x7fdc481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 1 -- 172.21.15.110:0/2064438574 >> v1:172.21.15.110:6789/0 conn(0x7fdc481430c0 legacy=0x7fdc4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 1 monclient: found mon.noname-b 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 1 -- 172.21.15.110:0/2064438574 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc480b44c0 con 0x7fdc480a98f0 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 1 -- 172.21.15.110:0/2064438574 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdc38001690 con 0x7fdc480a98f0 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc46ffd640 10 monclient: dump: 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:16.578 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:16.579 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.579 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:16.579 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:16.579 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:16.579 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:16.579 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:16.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc4ea06640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:44:16.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.578+0000 7fdc4ea06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:16.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.582+0000 7fdc46ffd640 1 -- 172.21.15.110:0/2064438574 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc38001860 con 0x7fdc480a98f0 2024-03-25T19:44:16.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.582+0000 7fdc46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:16.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.582+0000 7fdc4ea06640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:16.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.582+0000 7fdc4ea06640 4 set_mon_vals no callback set 2024-03-25T19:44:16.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.582+0000 7fdc4ea06640 10 monclient: shutdown 2024-03-25T19:44:16.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.582+0000 7fdc4ea06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:16.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.582+0000 7fdc4ea06640 1 -- 172.21.15.110:0/2064438574 >> v1:172.21.15.110:6790/0 conn(0x7fdc480a98f0 legacy=0x7fdc480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.583+0000 7fdc4ea06640 1 -- 172.21.15.110:0/2064438574 shutdown_connections 2024-03-25T19:44:16.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.583+0000 7fdc4ea06640 1 -- 172.21.15.110:0/2064438574 >> 172.21.15.110:0/2064438574 conn(0x7fdc480b7ea0 msgr2=0x7fdc480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:16.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.583+0000 7fdc4ea06640 1 -- 172.21.15.110:0/2064438574 shutdown_connections 2024-03-25T19:44:16.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.583+0000 7fdc4ea06640 1 -- 172.21.15.110:0/2064438574 wait complete. 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.583+0000 7fdc4ea06640 10 monclient: build_initial_monmap 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.583+0000 7fdc4ea06640 10 monclient: monmap: 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:16.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.584+0000 7fdc4ea06640 1 librados: starting msgr at 2024-03-25T19:44:16.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.584+0000 7fdc4ea06640 1 librados: starting objecter 2024-03-25T19:44:16.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.584+0000 7fdc4ea06640 1 Processor -- start 2024-03-25T19:44:16.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.584+0000 7fdc4ea06640 1 -- start start 2024-03-25T19:44:16.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.584+0000 7fdc4ea06640 1 librados: setting wanted keys 2024-03-25T19:44:16.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.584+0000 7fdc4ea06640 1 librados: calling monclient init 2024-03-25T19:44:16.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.584+0000 7fdc4ea06640 10 monclient: init 2024-03-25T19:44:16.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:16.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:16.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 10 monclient(hunting): picked mon.noname-a con 0x7fdc480a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:16.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 10 monclient(hunting): picked mon.noname-c con 0x7fdc4813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:16.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 10 monclient(hunting): picked mon.noname-b con 0x7fdc481430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:16.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc481da990 con 0x7fdc480a98f0 2024-03-25T19:44:16.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc480abde0 con 0x7fdc481430c0 2024-03-25T19:44:16.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc481dabe0 con 0x7fdc4813f710 2024-03-25T19:44:16.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:16.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.585+0000 7fdc4ea06640 10 monclient(hunting): authenticate will time out at 4427.726101s 2024-03-25T19:44:16.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.597+0000 7fdc4cfa3640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fdc481430c0 0x7fdc481e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:53798/0 (socket says 172.21.15.110:53798) 2024-03-25T19:44:16.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.597+0000 7fdc4cfa3640 1 -- 172.21.15.110:0/1223824501 learned_addr learned my addr 172.21.15.110:0/1223824501 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:16.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.598+0000 7fdc44ff9640 10 client.?.objecter ms_handle_connect 0x7fdc481430c0 2024-03-25T19:44:16.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.598+0000 7fdc44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.605+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396141047 0 0) 0x7fdc480abde0 con 0x7fdc481430c0 2024-03-25T19:44:16.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.605+0000 7fdc44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.605+0000 7fdc44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.605+0000 7fdc44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.605+0000 7fdc44ff9640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:44:16.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.605+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc34004eb0 con 0x7fdc481430c0 2024-03-25T19:44:16.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.612+0000 7fdc44ff9640 10 client.?.objecter ms_handle_connect 0x7fdc4813f710 2024-03-25T19:44:16.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.612+0000 7fdc44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.620+0000 7fdc44ff9640 10 client.?.objecter ms_handle_connect 0x7fdc480a98f0 2024-03-25T19:44:16.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.620+0000 7fdc44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.627+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690759996 0 0) 0x7fdc481dabe0 con 0x7fdc4813f710 2024-03-25T19:44:16.627 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.627+0000 7fdc44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.627 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.627+0000 7fdc44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.627 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.627+0000 7fdc44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.627 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.627+0000 7fdc44ff9640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:44:16.627 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.627+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc480abde0 con 0x7fdc4813f710 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.630+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 480276353 0 0) 0x7fdc34004eb0 con 0x7fdc481430c0 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.630+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc481dabe0 con 0x7fdc481430c0 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.630+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdc40001ad0 con 0x7fdc481430c0 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.630+0000 7fdc44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.630+0000 7fdc44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.630+0000 7fdc44ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:16.630 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.631 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:16.631 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:16.631 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:16.631 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:16.631 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:16.631 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.630+0000 7fdc44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.644+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083288238 0 0) 0x7fdc480abde0 con 0x7fdc4813f710 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.644+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc34004eb0 con 0x7fdc4813f710 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.644+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdc3c0018b0 con 0x7fdc4813f710 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.644+0000 7fdc44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.644+0000 7fdc44ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.644+0000 7fdc44ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:16.644 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:16.645 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.645 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:16.645 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:16.645 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:16.645 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:16.645 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:16.646 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.645+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320331575 0 0) 0x7fdc481da990 con 0x7fdc480a98f0 2024-03-25T19:44:16.646 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.645+0000 7fdc44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.646 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.645+0000 7fdc44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.646 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.645+0000 7fdc44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.646 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.645+0000 7fdc44ff9640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:44:16.646 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.645+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc480abde0 con 0x7fdc480a98f0 2024-03-25T19:44:16.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3030584827 0 0) 0x7fdc481dabe0 con 0x7fdc481430c0 2024-03-25T19:44:16.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 >> v1:172.21.15.110:6790/0 conn(0x7fdc4813f710 legacy=0x7fdc481db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 >> v1:172.21.15.110:6789/0 conn(0x7fdc480a98f0 legacy=0x7fdc48138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:16.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 1 monclient: found mon.b 2024-03-25T19:44:16.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:16.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:16.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc481db7f0 con 0x7fdc481430c0 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 10 monclient: _renew_subs 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc48137b10 con 0x7fdc481430c0 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 10 monclient: _renew_subs 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc481353e0 con 0x7fdc481430c0 2024-03-25T19:44:16.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.654+0000 7fdc4ea06640 1 librados: init done 2024-03-25T19:44:16.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.655+0000 7fdc4ea06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.655+0000 7fdc4ea06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.655+0000 7fdc4ea06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:16.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.655+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc481dad70 con 0x7fdc481430c0 2024-03-25T19:44:16.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.930+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc40001fe0 con 0x7fdc481430c0 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.930+0000 7fdc44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.930+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdc40004d80 con 0x7fdc481430c0 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.930+0000 7fdc44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.930+0000 7fdc44ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.930+0000 7fdc44ff9640 10 monclient: dump: 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:16.931 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:16.932 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:16.932 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:16.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.930+0000 7fdc467fc640 4 set_mon_vals no callback set 2024-03-25T19:44:16.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.931+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fdc4001ad50 con 0x7fdc481430c0 2024-03-25T19:44:16.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.932+0000 7fdc47fff640 10 monclient: get_auth_request con 0x7fdc3406df00 auth_method 0 2024-03-25T19:44:16.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.933+0000 7fdc44ff9640 10 client.4402.objecter ms_handle_connect 0x7fdc3406df00 2024-03-25T19:44:16.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.936+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fdc40054a80 con 0x7fdc481430c0 2024-03-25T19:44:16.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.936+0000 7fdc44ff9640 10 client.4402.objecter ms_dispatch 0x7fdc48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:16.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.936+0000 7fdc44ff9640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:16.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.936+0000 7fdc44ff9640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:16.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.936+0000 7fdc44ff9640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T19:44:16.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.936+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc400503c0 con 0x7fdc481430c0 2024-03-25T19:44:16.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.936+0000 7fdc44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:16.936+0000 7fdc44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:17.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.156+0000 7fdc4ea06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:17.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.157+0000 7fdc4ea06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:17.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.157+0000 7fdc4ea06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:17.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.157+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc480627b0 con 0x7fdc481430c0 2024-03-25T19:44:17.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.186+0000 7fdc44ff9640 1 -- 172.21.15.110:0/1223824501 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fdc400524c0 con 0x7fdc481430c0 2024-03-25T19:44:17.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.186+0000 7fdc44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:17.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.186+0000 7fdc44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:17.186 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:44:17.186 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:17.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 10 librados: watch_flush enter 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 10 librados: watch_flush exit 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 10 client.4402.objecter successfully canceled tick 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 >> v1:172.21.15.178:6800/1132739058 conn(0x7fdc3406df00 legacy=0x7fdc340703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 10 monclient: shutdown 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 >> v1:172.21.15.178:6789/0 conn(0x7fdc481430c0 legacy=0x7fdc481e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 shutdown_connections 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.188+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 >> 172.21.15.110:0/1223824501 conn(0x7fdc480b7ea0 msgr2=0x7fdc48074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.189+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 shutdown_connections 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.189+0000 7fdc4ea06640 1 -- 172.21.15.110:0/1223824501 wait complete. 2024-03-25T19:44:17.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:17.189+0000 7fdc4ea06640 1 librados: shutdown 2024-03-25T19:44:17.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:24.007 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:24.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.122+0000 7f0328855640 10 monclient: get_monmap_and_config 2024-03-25T19:44:24.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.122+0000 7f0328855640 10 monclient: build_initial_monmap 2024-03-25T19:44:24.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.122+0000 7f0328855640 10 monclient: monmap: 2024-03-25T19:44:24.122 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:24.122 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:24.123 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:44:24.123888+0000 2024-03-25T19:44:24.123 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:44:24.123888+0000 2024-03-25T19:44:24.123 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:24.123 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:24.123 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:24.123 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:44:24.123 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:44:24.123 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:24.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.123+0000 7f0328855640 1 Processor -- start 2024-03-25T19:44:24.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.123+0000 7f0328855640 1 -- start start 2024-03-25T19:44:24.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.123+0000 7f0328855640 10 monclient: init 2024-03-25T19:44:24.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 10 monclient(hunting): picked mon.noname-a con 0x7f03240b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 10 monclient(hunting): picked mon.noname-c con 0x7f03240b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 10 monclient(hunting): picked mon.noname-b con 0x7f032414be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240af940 con 0x7f03240b4d60 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240b0a70 con 0x7f03240b1730 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240b0d50 con 0x7f032414be90 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:24.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.124+0000 7f0328855640 10 monclient(hunting): authenticate will time out at 4435.265801s 2024-03-25T19:44:24.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.133+0000 7f0322d9d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f032414be90 0x7f032414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:50122/0 (socket says 172.21.15.110:50122) 2024-03-25T19:44:24.133 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.133+0000 7f0322d9d640 1 -- 172.21.15.110:0/3155370180 learned_addr learned my addr 172.21.15.110:0/3155370180 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:24.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.155+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444679001 0 0) 0x7f03240b0d50 con 0x7f032414be90 2024-03-25T19:44:24.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.155+0000 7f032159a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.155+0000 7f032159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.155+0000 7f032159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.155+0000 7f032159a640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:44:24.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.155+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030c003520 con 0x7f032414be90 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.167+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844754420 0 0) 0x7f03240af940 con 0x7f03240b4d60 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.167+0000 7f032159a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.167+0000 7f032159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.167+0000 7f032159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.167+0000 7f032159a640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.167+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03240b0d50 con 0x7f03240b4d60 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.172+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442902137 0 0) 0x7f03240b0a70 con 0x7f03240b1730 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.172+0000 7f032159a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.172+0000 7f032159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.172+0000 7f032159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.172+0000 7f032159a640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.172+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03240af940 con 0x7f03240b1730 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786018055 0 0) 0x7f030c003520 con 0x7f032414be90 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 >> v1:172.21.15.178:6789/0 conn(0x7f03240b1730 legacy=0x7f03240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 >> v1:172.21.15.110:6789/0 conn(0x7f03240b4d60 legacy=0x7f032413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 10 monclient: _finish_hunting 0 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 1 monclient: found mon.noname-b 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03240b1030 con 0x7f032414be90 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 10 monclient: _finish_auth 0 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 10 monclient: _check_auth_tickets 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:24.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f031c001a10 con 0x7f032414be90 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f032159a640 10 monclient: dump: 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f0328855640 5 monclient: authenticate success, global_id 4400 2024-03-25T19:44:24.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.174+0000 7f0328855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.187+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f031c001b70 con 0x7f032414be90 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.187+0000 7f032159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.187+0000 7f032159a640 1 -- 172.21.15.110:0/3155370180 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f031c001f80 con 0x7f032414be90 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.187+0000 7f0328855640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.187+0000 7f0328855640 4 set_mon_vals no callback set 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.187+0000 7f032159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.187+0000 7f032159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.187+0000 7f032159a640 10 monclient: dump: 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:24.187 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.188 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.188 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.188 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:24.188 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:24.188 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:24.188 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:24.188 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:24.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.188+0000 7f0328855640 10 monclient: shutdown 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.188+0000 7f0328855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.188+0000 7f0328855640 1 -- 172.21.15.110:0/3155370180 >> v1:172.21.15.110:6790/0 conn(0x7f032414be90 legacy=0x7f032414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.188+0000 7f0328855640 1 -- 172.21.15.110:0/3155370180 shutdown_connections 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.188+0000 7f0328855640 1 -- 172.21.15.110:0/3155370180 >> 172.21.15.110:0/3155370180 conn(0x7f03240617a0 msgr2=0x7f0324061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.188+0000 7f0328855640 1 -- 172.21.15.110:0/3155370180 shutdown_connections 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.188+0000 7f0328855640 1 -- 172.21.15.110:0/3155370180 wait complete. 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.189+0000 7f0328855640 10 monclient: build_initial_monmap 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.189+0000 7f0328855640 10 monclient: monmap: 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:44:24.190 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 librados: starting msgr at 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 librados: starting objecter 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 Processor -- start 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 -- start start 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 librados: setting wanted keys 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 librados: calling monclient init 2024-03-25T19:44:24.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 10 monclient: init 2024-03-25T19:44:24.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:24.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:24.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 10 monclient(hunting): picked mon.noname-c con 0x7f03240b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:24.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 10 monclient(hunting): picked mon.noname-a con 0x7f03240b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:24.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 10 monclient(hunting): picked mon.noname-b con 0x7f032414be90 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:24.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324140380 con 0x7f03240b4d60 2024-03-25T19:44:24.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324142120 con 0x7f032414be90 2024-03-25T19:44:24.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324142400 con 0x7f03240b1730 2024-03-25T19:44:24.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:24.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.190+0000 7f0328855640 10 monclient(hunting): authenticate will time out at 4435.331711s 2024-03-25T19:44:24.199 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.199+0000 7f0322d9d640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f032414be90 0x7f03241e2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:40442/0 (socket says 172.21.15.110:40442) 2024-03-25T19:44:24.199 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.199+0000 7f0322d9d640 1 -- 172.21.15.110:0/4189524819 learned_addr learned my addr 172.21.15.110:0/4189524819 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:24.204 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.205+0000 7f0312ffd640 10 client.?.objecter ms_handle_connect 0x7f032414be90 2024-03-25T19:44:24.204 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.205+0000 7f0312ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.209+0000 7f0312ffd640 10 client.?.objecter ms_handle_connect 0x7f03240b1730 2024-03-25T19:44:24.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.209+0000 7f0312ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.213+0000 7f0312ffd640 10 client.?.objecter ms_handle_connect 0x7f03240b4d60 2024-03-25T19:44:24.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.213+0000 7f0312ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.218+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559989744 0 0) 0x7f0324142120 con 0x7f032414be90 2024-03-25T19:44:24.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.218+0000 7f0312ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.218+0000 7f0312ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.218+0000 7f0312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.218+0000 7f0312ffd640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:44:24.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.218+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304004eb0 con 0x7f032414be90 2024-03-25T19:44:24.230 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.231+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848018895 0 0) 0x7f0324140380 con 0x7f03240b4d60 2024-03-25T19:44:24.231 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.231+0000 7f0312ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.231 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.231+0000 7f0312ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.231 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.231+0000 7f0312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.231 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.231+0000 7f0312ffd640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:44:24.231 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.231+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0324142120 con 0x7f03240b4d60 2024-03-25T19:44:24.233 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.233+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3490847433 0 0) 0x7f0304004eb0 con 0x7f032414be90 2024-03-25T19:44:24.233 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.233+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0324140380 con 0x7f032414be90 2024-03-25T19:44:24.233 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.233+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f031c001ec0 con 0x7f032414be90 2024-03-25T19:44:24.233 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.233+0000 7f0312ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.233+0000 7f0312ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.233+0000 7f0312ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.233+0000 7f0312ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:24.234 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.234+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490172717 0 0) 0x7f0324142400 con 0x7f03240b1730 2024-03-25T19:44:24.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.234+0000 7f0312ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.234+0000 7f0312ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.234+0000 7f0312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.234+0000 7f0312ffd640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:44:24.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.234+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304004eb0 con 0x7f03240b1730 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.251+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 682081408 0 0) 0x7f0324142120 con 0x7f03240b4d60 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.251+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0324142400 con 0x7f03240b4d60 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.251+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0318001870 con 0x7f03240b4d60 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.251+0000 7f0312ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.251+0000 7f0312ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.251+0000 7f0312ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.251 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:24.252 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:24.252 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:24.252 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:24.252 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2604712757 0 0) 0x7f0324140380 con 0x7f032414be90 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 >> v1:172.21.15.110:6790/0 conn(0x7f03240b1730 legacy=0x7f0324136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 >> v1:172.21.15.110:6789/0 conn(0x7f03240b4d60 legacy=0x7f0324136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 1 monclient: found mon.b 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03241426e0 con 0x7f032414be90 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.254+0000 7f0312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 5 monclient: authenticate success, global_id 4408 2024-03-25T19:44:24.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 10 monclient: _renew_subs 2024-03-25T19:44:24.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:24.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03241429c0 con 0x7f032414be90 2024-03-25T19:44:24.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:24.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 10 monclient: _renew_subs 2024-03-25T19:44:24.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:24.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0324140cd0 con 0x7f032414be90 2024-03-25T19:44:24.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.255+0000 7f0328855640 1 librados: init done 2024-03-25T19:44:24.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.256+0000 7f0328855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.256+0000 7f0328855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.256+0000 7f0328855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:24.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.256+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0324072740 con 0x7f032414be90 2024-03-25T19:44:24.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.536+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f031c004e10 con 0x7f032414be90 2024-03-25T19:44:24.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.536+0000 7f0312ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.536+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f031c0015d0 con 0x7f032414be90 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.536+0000 7f0312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.536+0000 7f0312ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.536+0000 7f0312ffd640 10 monclient: dump: 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:24.536 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.536+0000 7f0320d99640 4 set_mon_vals no callback set 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.536+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f031c01af90 con 0x7f032414be90 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.537+0000 7f032259c640 10 monclient: get_auth_request con 0x7f030406df70 auth_method 0 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.537+0000 7f0312ffd640 10 client.4408.objecter ms_handle_connect 0x7f030406df70 2024-03-25T19:44:24.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.542+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f031c054b50 con 0x7f032414be90 2024-03-25T19:44:24.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.542+0000 7f0312ffd640 10 client.4408.objecter ms_dispatch 0x7f0324000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:24.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.542+0000 7f0312ffd640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:24.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.542+0000 7f0312ffd640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:24.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.542+0000 7f0312ffd640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T19:44:24.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.542+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f031c050490 con 0x7f032414be90 2024-03-25T19:44:24.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.542+0000 7f0312ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.542+0000 7f0312ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:24.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.789+0000 7f0328855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.789+0000 7f0328855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.789+0000 7f0328855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:24.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.789+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03240627b0 con 0x7f032414be90 2024-03-25T19:44:24.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.954+0000 7f0312ffd640 1 -- 172.21.15.110:0/4189524819 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f031c052590 con 0x7f032414be90 2024-03-25T19:44:24.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.954+0000 7f0312ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.954+0000 7f0312ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:24.955 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:44:24.955 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:24.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 10 librados: watch_flush enter 2024-03-25T19:44:24.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 10 librados: watch_flush exit 2024-03-25T19:44:24.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T19:44:24.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 10 client.4408.objecter successfully canceled tick 2024-03-25T19:44:24.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 >> v1:172.21.15.178:6800/1132739058 conn(0x7f030406df70 legacy=0x7f0304070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 10 monclient: shutdown 2024-03-25T19:44:24.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:24.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 >> v1:172.21.15.178:6789/0 conn(0x7f032414be90 legacy=0x7f03241e2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 shutdown_connections 2024-03-25T19:44:24.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 >> 172.21.15.110:0/4189524819 conn(0x7f03240617a0 msgr2=0x7f032413c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:24.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 shutdown_connections 2024-03-25T19:44:24.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 1 -- 172.21.15.110:0/4189524819 wait complete. 2024-03-25T19:44:24.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:24.957+0000 7f0328855640 1 librados: shutdown 2024-03-25T19:44:24.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:31.777 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:31.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.891+0000 7f657ed9d640 10 monclient: get_monmap_and_config 2024-03-25T19:44:31.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.891+0000 7f657ed9d640 10 monclient: build_initial_monmap 2024-03-25T19:44:31.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.891+0000 7f657ed9d640 10 monclient: monmap: 2024-03-25T19:44:31.916 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:31.916 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:31.916 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:44:31.892525+0000 2024-03-25T19:44:31.916 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:44:31.892525+0000 2024-03-25T19:44:31.917 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:31.917 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:31.917 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:31.917 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:44:31.917 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:44:31.918 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.892+0000 7f657ed9d640 1 Processor -- start 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.892+0000 7f657ed9d640 1 -- start start 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.892+0000 7f657ed9d640 10 monclient: init 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.892+0000 7f657ed9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f65780a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f657813f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f65781430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6578142520 con 0x7f657813f710 2024-03-25T19:44:31.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65780b28a0 con 0x7f65780a98f0 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65780b2a30 con 0x7f65781430c0 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.893+0000 7f657ed9d640 10 monclient(hunting): authenticate will time out at 4443.034029s 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.900+0000 7f6577fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f657813f710 0x7f65781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42912/0 (socket says 172.21.15.110:42912) 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.900+0000 7f6577fff640 1 -- 172.21.15.110:0/4270247729 learned_addr learned my addr 172.21.15.110:0/4270247729 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.906+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628548712 0 0) 0x7f6578142520 con 0x7f657813f710 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.906+0000 7f65777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.906+0000 7f65777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.906+0000 7f65777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.906+0000 7f65777fe640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.906+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6560003520 con 0x7f657813f710 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.921+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049627548 0 0) 0x7f65780b2a30 con 0x7f65781430c0 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.921+0000 7f65777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.921+0000 7f65777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.921+0000 7f65777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.921+0000 7f65777fe640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.921+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6578142520 con 0x7f65781430c0 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176215542 0 0) 0x7f6560003520 con 0x7f657813f710 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 >> v1:172.21.15.110:6790/0 conn(0x7f65781430c0 legacy=0x7f657814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 >> v1:172.21.15.178:6789/0 conn(0x7f65780a98f0 legacy=0x7f65780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 1 monclient: found mon.noname-a 2024-03-25T19:44:31.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:31.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:44:31.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65780b44c0 con 0x7f657813f710 2024-03-25T19:44:31.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:31.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f656c001690 con 0x7f657813f710 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f657ed9d640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f657ed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f65777fe640 10 monclient: dump: 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:31.928 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:31.929 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:31.929 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:31.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.922+0000 7f657ed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:31.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.930+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f656c0017f0 con 0x7f657813f710 2024-03-25T19:44:31.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.930+0000 7f65777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:31.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.930+0000 7f65777fe640 1 -- 172.21.15.110:0/4270247729 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f656c001c00 con 0x7f657813f710 2024-03-25T19:44:31.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.930+0000 7f657ed9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:31.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.930+0000 7f657ed9d640 4 set_mon_vals no callback set 2024-03-25T19:44:31.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.930+0000 7f65777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.930+0000 7f65777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.930+0000 7f65777fe640 10 monclient: dump: 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:31.931 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:31.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.931+0000 7f657ed9d640 10 monclient: shutdown 2024-03-25T19:44:31.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.931+0000 7f657ed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:31.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.931+0000 7f657ed9d640 1 -- 172.21.15.110:0/4270247729 >> v1:172.21.15.110:6789/0 conn(0x7f657813f710 legacy=0x7f65781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.931+0000 7f657ed9d640 1 -- 172.21.15.110:0/4270247729 shutdown_connections 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.931+0000 7f657ed9d640 1 -- 172.21.15.110:0/4270247729 >> 172.21.15.110:0/4270247729 conn(0x7f65780b7ea0 msgr2=0x7f65780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.931+0000 7f657ed9d640 1 -- 172.21.15.110:0/4270247729 shutdown_connections 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.931+0000 7f657ed9d640 1 -- 172.21.15.110:0/4270247729 wait complete. 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 10 monclient: build_initial_monmap 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 10 monclient: monmap: 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:31.933 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 1 librados: starting msgr at 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 1 librados: starting objecter 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 1 Processor -- start 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 1 -- start start 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 1 librados: setting wanted keys 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 1 librados: calling monclient init 2024-03-25T19:44:31.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.932+0000 7f657ed9d640 10 monclient: init 2024-03-25T19:44:31.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:31.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:31.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f65780a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:31.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f657813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:31.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f65781430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:31.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6578137f10 con 0x7f65781430c0 2024-03-25T19:44:31.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65781dc070 con 0x7f657813f710 2024-03-25T19:44:31.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6578137100 con 0x7f65780a98f0 2024-03-25T19:44:31.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:31.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.933+0000 7f657ed9d640 10 monclient(hunting): authenticate will time out at 4443.074819s 2024-03-25T19:44:31.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.948+0000 7f657cb39640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f65780a98f0 0x7f6578137800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33486/0 (socket says 172.21.15.110:33486) 2024-03-25T19:44:31.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.948+0000 7f657cb39640 1 -- 172.21.15.110:0/72947406 learned_addr learned my addr 172.21.15.110:0/72947406 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:31.948 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.949+0000 7f65757fa640 10 client.?.objecter ms_handle_connect 0x7f65780a98f0 2024-03-25T19:44:31.948 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.949+0000 7f65757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.951+0000 7f65757fa640 10 client.?.objecter ms_handle_connect 0x7f657813f710 2024-03-25T19:44:31.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.951+0000 7f65757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.955+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431888423 0 0) 0x7f6578137100 con 0x7f65780a98f0 2024-03-25T19:44:31.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.956+0000 7f65757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.956+0000 7f65757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.956+0000 7f65757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.956+0000 7f65757fa640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:44:31.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.956+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6564004eb0 con 0x7f65780a98f0 2024-03-25T19:44:31.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.956+0000 7f65757fa640 10 client.?.objecter ms_handle_connect 0x7f65781430c0 2024-03-25T19:44:31.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.956+0000 7f65757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.965+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897439385 0 0) 0x7f65781dc070 con 0x7f657813f710 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.965+0000 7f65757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.965+0000 7f65757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.965+0000 7f65757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.965+0000 7f65757fa640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.965+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6578137100 con 0x7f657813f710 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.973+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4001894432 0 0) 0x7f6578137100 con 0x7f657813f710 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.973+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65781dc070 con 0x7f657813f710 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.973+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f656c001b00 con 0x7f657813f710 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.973+0000 7f65757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.973+0000 7f65757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.973+0000 7f65757fa640 10 monclient(hunting): dump: 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:31.974 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:31.974 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:31.974 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:31.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.973+0000 7f65757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:31.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.974+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790147413 0 0) 0x7f6578137f10 con 0x7f65781430c0 2024-03-25T19:44:31.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.974+0000 7f65757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.974+0000 7f65757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.974+0000 7f65757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.974+0000 7f65757fa640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:44:31.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.974+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6578137100 con 0x7f65781430c0 2024-03-25T19:44:31.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.987+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046340926 0 0) 0x7f6564004eb0 con 0x7f65780a98f0 2024-03-25T19:44:31.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.987+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6578137f10 con 0x7f65780a98f0 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.987+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f65680019d0 con 0x7f65780a98f0 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.987+0000 7f65757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.987+0000 7f65757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.987+0000 7f65757fa640 10 monclient(hunting): dump: 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:31.987 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:31.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1389808070 0 0) 0x7f65781dc070 con 0x7f657813f710 2024-03-25T19:44:31.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 >> v1:172.21.15.110:6790/0 conn(0x7f65780a98f0 legacy=0x7f6578137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 >> v1:172.21.15.110:6789/0 conn(0x7f65781430c0 legacy=0x7f65781db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:31.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 1 monclient: found mon.b 2024-03-25T19:44:31.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65781373e0 con 0x7f657813f710 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f65757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 5 monclient: authenticate success, global_id 4414 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 10 monclient: _renew_subs 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65781358e0 con 0x7f657813f710 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 10 monclient: _renew_subs 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6578136150 con 0x7f657813f710 2024-03-25T19:44:31.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.990+0000 7f657ed9d640 1 librados: init done 2024-03-25T19:44:31.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.991+0000 7f657ed9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.991+0000 7f657ed9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.991+0000 7f657ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:31.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:31.991+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65780627b0 con 0x7f657813f710 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.271+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f656c004a90 con 0x7f657813f710 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.271+0000 7f65757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.271+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f656c001cc0 con 0x7f657813f710 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.271+0000 7f65757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.271+0000 7f65757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.271+0000 7f65757fa640 10 monclient: dump: 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:32.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.271+0000 7f6576ffd640 4 set_mon_vals no callback set 2024-03-25T19:44:32.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.272+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f656c01ac10 con 0x7f657813f710 2024-03-25T19:44:32.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.273+0000 7f657cb39640 10 monclient: get_auth_request con 0x7f656406dfb0 auth_method 0 2024-03-25T19:44:32.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.273+0000 7f65757fa640 10 client.4414.objecter ms_handle_connect 0x7f656406dfb0 2024-03-25T19:44:32.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.277+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f656c054850 con 0x7f657813f710 2024-03-25T19:44:32.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.277+0000 7f65757fa640 10 client.4414.objecter ms_dispatch 0x7f6578000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:32.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.277+0000 7f65757fa640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:32.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.277+0000 7f65757fa640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:32.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.277+0000 7f65757fa640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T19:44:32.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.278+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f656c050190 con 0x7f657813f710 2024-03-25T19:44:32.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.278+0000 7f65757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:32.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.278+0000 7f65757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:32.529 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.529+0000 7f657ed9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.529 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.529+0000 7f657ed9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.529 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.529+0000 7f657ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:32.529 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.529+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f657813e250 con 0x7f657813f710 2024-03-25T19:44:32.555 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.556+0000 7f65757fa640 1 -- 172.21.15.110:0/72947406 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f656c052290 con 0x7f657813f710 2024-03-25T19:44:32.556 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.556+0000 7f65757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.556 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.556+0000 7f65757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:32.556 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:44:32.556 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:32.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 10 librados: watch_flush enter 2024-03-25T19:44:32.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 10 librados: watch_flush exit 2024-03-25T19:44:32.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 10 client.4414.objecter successfully canceled tick 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 >> v1:172.21.15.178:6800/1132739058 conn(0x7f656406dfb0 legacy=0x7f6564070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 10 monclient: shutdown 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 >> v1:172.21.15.178:6789/0 conn(0x7f657813f710 legacy=0x7f65781db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 shutdown_connections 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 >> 172.21.15.110:0/72947406 conn(0x7f65780b7ea0 msgr2=0x7f657814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 shutdown_connections 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.558+0000 7f657ed9d640 1 -- 172.21.15.110:0/72947406 wait complete. 2024-03-25T19:44:32.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:32.559+0000 7f657ed9d640 1 librados: shutdown 2024-03-25T19:44:32.569 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:37.548 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:44.832 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.561 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:44.845 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.600 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:44.885 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.605 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:44.859 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.615 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:44.870 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.622 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:44.905 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.625 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:44.910 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.649 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:04:44.904 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.659 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:44.943 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.375 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:39.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.487+0000 7f66af7e8640 10 monclient: get_monmap_and_config 2024-03-25T19:44:39.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.487+0000 7f66af7e8640 10 monclient: build_initial_monmap 2024-03-25T19:44:39.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.487+0000 7f66af7e8640 10 monclient: monmap: 2024-03-25T19:44:39.487 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:39.487 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:39.487 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:44:39.488589+0000 2024-03-25T19:44:39.487 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:44:39.488589+0000 2024-03-25T19:44:39.487 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:39.488 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:39.488 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:39.488 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:44:39.488 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:44:39.488 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.488+0000 7f66af7e8640 1 Processor -- start 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.488+0000 7f66af7e8640 1 -- start start 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.488+0000 7f66af7e8640 10 monclient: init 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 10 monclient(hunting): picked mon.noname-c con 0x7f66a80a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 10 monclient(hunting): picked mon.noname-a con 0x7f66a813f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 10 monclient(hunting): picked mon.noname-b con 0x7f66a81430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:39.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a8142520 con 0x7f66a813f710 2024-03-25T19:44:39.490 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a80b28a0 con 0x7f66a80a98f0 2024-03-25T19:44:39.490 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a80b2a30 con 0x7f66a81430c0 2024-03-25T19:44:39.490 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:39.490 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.489+0000 7f66af7e8640 10 monclient(hunting): authenticate will time out at 4450.630357s 2024-03-25T19:44:39.493 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.493+0000 7f66ad584640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f66a80a98f0 0x7f66a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:59182/0 (socket says 172.21.15.110:59182) 2024-03-25T19:44:39.493 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.493+0000 7f66ad584640 1 -- 172.21.15.110:0/813824578 learned_addr learned my addr 172.21.15.110:0/813824578 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:39.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.507+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430848186 0 0) 0x7f66a8142520 con 0x7f66a813f710 2024-03-25T19:44:39.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.507+0000 7f669ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.507+0000 7f669ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.507+0000 7f669ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.507+0000 7f669ffff640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:44:39.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.507+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6694003520 con 0x7f66a813f710 2024-03-25T19:44:39.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.509+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408016427 0 0) 0x7f66a80b28a0 con 0x7f66a80a98f0 2024-03-25T19:44:39.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.510+0000 7f669ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.510+0000 7f669ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.510+0000 7f669ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.510+0000 7f669ffff640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:44:39.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.510+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a8142520 con 0x7f66a80a98f0 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2018015453 0 0) 0x7f6694003520 con 0x7f66a813f710 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 >> v1:172.21.15.110:6790/0 conn(0x7f66a81430c0 legacy=0x7f66a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 >> v1:172.21.15.178:6789/0 conn(0x7f66a80a98f0 legacy=0x7f66a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 1 monclient: found mon.noname-a 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a80b44c0 con 0x7f66a813f710 2024-03-25T19:44:39.513 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 10 monclient: _finish_auth 0 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6698001690 con 0x7f66a813f710 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f66af7e8640 5 monclient: authenticate success, global_id 4407 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f66af7e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f669ffff640 10 monclient: dump: 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:39.514 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.515 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:39.515 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:39.515 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:39.515 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:39.515 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:39.515 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.513+0000 7f66af7e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:39.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.519+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66980017f0 con 0x7f66a813f710 2024-03-25T19:44:39.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.519+0000 7f669ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.519+0000 7f669ffff640 1 -- 172.21.15.110:0/813824578 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6698001c00 con 0x7f66a813f710 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.519+0000 7f669ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.519+0000 7f669ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.519+0000 7f669ffff640 10 monclient: dump: 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:39.519 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:39.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.519+0000 7f66af7e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:39.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.519+0000 7f66af7e8640 4 set_mon_vals no callback set 2024-03-25T19:44:39.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 10 monclient: shutdown 2024-03-25T19:44:39.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:39.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 1 -- 172.21.15.110:0/813824578 >> v1:172.21.15.110:6789/0 conn(0x7f66a813f710 legacy=0x7f66a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 1 -- 172.21.15.110:0/813824578 shutdown_connections 2024-03-25T19:44:39.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 1 -- 172.21.15.110:0/813824578 >> 172.21.15.110:0/813824578 conn(0x7f66a80b7ea0 msgr2=0x7f66a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:39.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 1 -- 172.21.15.110:0/813824578 shutdown_connections 2024-03-25T19:44:39.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 1 -- 172.21.15.110:0/813824578 wait complete. 2024-03-25T19:44:39.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 10 monclient: build_initial_monmap 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.520+0000 7f66af7e8640 10 monclient: monmap: 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.521+0000 7f66af7e8640 1 librados: starting msgr at 2024-03-25T19:44:39.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.521+0000 7f66af7e8640 1 librados: starting objecter 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.521+0000 7f66af7e8640 1 Processor -- start 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.521+0000 7f66af7e8640 1 -- start start 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.521+0000 7f66af7e8640 1 librados: setting wanted keys 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.521+0000 7f66af7e8640 1 librados: calling monclient init 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.521+0000 7f66af7e8640 10 monclient: init 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 10 monclient(hunting): picked mon.noname-a con 0x7f66a80a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 10 monclient(hunting): picked mon.noname-c con 0x7f66a813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 10 monclient(hunting): picked mon.noname-b con 0x7f66a81430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a8137080 con 0x7f66a80a98f0 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a8137360 con 0x7f66a81430c0 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a8135b10 con 0x7f66a813f710 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:39.524 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.522+0000 7f66af7e8640 10 monclient(hunting): authenticate will time out at 4450.663219s 2024-03-25T19:44:39.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.524+0000 7f66acd83640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f66a813f710 0x7f66a80b6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:38190/0 (socket says 172.21.15.110:38190) 2024-03-25T19:44:39.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.524+0000 7f66acd83640 1 -- 172.21.15.110:0/2348595397 learned_addr learned my addr 172.21.15.110:0/2348595397 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:39.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.526+0000 7f669dffb640 10 client.?.objecter ms_handle_connect 0x7f66a813f710 2024-03-25T19:44:39.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.526+0000 7f669dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.531+0000 7f669dffb640 10 client.?.objecter ms_handle_connect 0x7f66a81430c0 2024-03-25T19:44:39.531 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.531+0000 7f669dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.536+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614399043 0 0) 0x7f66a8135b10 con 0x7f66a813f710 2024-03-25T19:44:39.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.536+0000 7f669dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.536+0000 7f669dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.536+0000 7f669dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.536+0000 7f669dffb640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:44:39.536 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.536+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668c004eb0 con 0x7f66a813f710 2024-03-25T19:44:39.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.542+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553856052 0 0) 0x7f66a8137360 con 0x7f66a81430c0 2024-03-25T19:44:39.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.542+0000 7f669dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.542+0000 7f669dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.542+0000 7f669dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.542+0000 7f669dffb640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:44:39.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.542+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a8135b10 con 0x7f66a81430c0 2024-03-25T19:44:39.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.548+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1121718660 0 0) 0x7f668c004eb0 con 0x7f66a813f710 2024-03-25T19:44:39.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.548+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66a8137360 con 0x7f66a813f710 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.549+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f66980035f0 con 0x7f66a813f710 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.549+0000 7f669dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.549+0000 7f669dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.549+0000 7f669dffb640 10 monclient(hunting): dump: 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:39.549 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:39.550 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:39.550 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:39.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.549+0000 7f669dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:39.556 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 292508498 0 0) 0x7f66a8137360 con 0x7f66a813f710 2024-03-25T19:44:39.556 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 >> v1:172.21.15.178:6789/0 conn(0x7f66a81430c0 legacy=0x7f66a8130340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 >> v1:172.21.15.110:6789/0 conn(0x7f66a80a98f0 legacy=0x7f66a8137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 1 monclient: found mon.c 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8135df0 con 0x7f66a813f710 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f669dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f66af7e8640 5 monclient: authenticate success, global_id 4415 2024-03-25T19:44:39.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f66af7e8640 10 monclient: _renew_subs 2024-03-25T19:44:39.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f66af7e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:44:39.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.556+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a8136340 con 0x7f66a813f710 2024-03-25T19:44:39.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.557+0000 7f66af7e8640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:39.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.557+0000 7f66af7e8640 10 monclient: _renew_subs 2024-03-25T19:44:39.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.557+0000 7f66af7e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:44:39.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.557+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a81352e0 con 0x7f66a813f710 2024-03-25T19:44:39.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.557+0000 7f66af7e8640 1 librados: init done 2024-03-25T19:44:39.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.558+0000 7f66af7e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.558+0000 7f66af7e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.558+0000 7f66af7e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:44:39.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.558+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a80627b0 con 0x7f66a813f710 2024-03-25T19:44:39.838 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.839+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6698003b40 con 0x7f66a813f710 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.839+0000 7f669dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.839+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6698003f50 con 0x7f66a813f710 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.839+0000 7f669dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.839+0000 7f669dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.839+0000 7f669dffb640 10 monclient: dump: 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.839 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:39.840 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:39.840 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:39.840 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:39.840 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:39.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.839+0000 7f669f7fe640 4 set_mon_vals no callback set 2024-03-25T19:44:39.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.840+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f6698004170 con 0x7f66a813f710 2024-03-25T19:44:39.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.840+0000 7f66ad584640 10 monclient: get_auth_request con 0x7f668c06dd00 auth_method 0 2024-03-25T19:44:39.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.841+0000 7f669dffb640 10 client.4415.objecter ms_handle_connect 0x7f668c06dd00 2024-03-25T19:44:39.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.846+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f6698054a50 con 0x7f66a813f710 2024-03-25T19:44:39.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.846+0000 7f669dffb640 10 client.4415.objecter ms_dispatch 0x7f66a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:39.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.846+0000 7f669dffb640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:39.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.846+0000 7f669dffb640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:39.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.846+0000 7f669dffb640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-25T19:44:39.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.846+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6698050560 con 0x7f66a813f710 2024-03-25T19:44:39.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.846+0000 7f669dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:39.846+0000 7f669dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:40.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.118+0000 7f66af7e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:40.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.118+0000 7f66af7e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:40.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.118+0000 7f66af7e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:44:40.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.118+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66a814f110 con 0x7f66a813f710 2024-03-25T19:44:40.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.145+0000 7f669dffb640 1 -- 172.21.15.110:0/2348595397 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f6698052490 con 0x7f66a813f710 2024-03-25T19:44:40.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.145+0000 7f669dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:40.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.145+0000 7f669dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:40.145 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:44:40.145 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.147+0000 7f66af7e8640 10 librados: watch_flush enter 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.147+0000 7f66af7e8640 10 librados: watch_flush exit 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.147+0000 7f66af7e8640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.147+0000 7f66af7e8640 10 client.4415.objecter successfully canceled tick 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.147+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 >> v1:172.21.15.178:6800/1132739058 conn(0x7f668c06dd00 legacy=0x7f668c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.147+0000 7f66af7e8640 10 monclient: shutdown 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.147+0000 7f66af7e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.147+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 >> v1:172.21.15.110:6790/0 conn(0x7f66a813f710 legacy=0x7f66a80b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.148+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 shutdown_connections 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.148+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 >> 172.21.15.110:0/2348595397 conn(0x7f66a80b7ea0 msgr2=0x7f66a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.148+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 shutdown_connections 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.148+0000 7f66af7e8640 1 -- 172.21.15.110:0/2348595397 wait complete. 2024-03-25T19:44:40.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:40.148+0000 7f66af7e8640 1 librados: shutdown 2024-03-25T19:44:40.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:46.966 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:47.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.079+0000 7f231ba6e640 10 monclient: get_monmap_and_config 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.079+0000 7f231ba6e640 10 monclient: build_initial_monmap 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.079+0000 7f231ba6e640 10 monclient: monmap: 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:44:47.081219+0000 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:44:47.081219+0000 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:44:47.080 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.080+0000 7f231ba6e640 1 Processor -- start 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.080+0000 7f231ba6e640 1 -- start start 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.080+0000 7f231ba6e640 10 monclient: init 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.080+0000 7f231ba6e640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.080+0000 7f231ba6e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.080+0000 7f231ba6e640 10 monclient(hunting): picked mon.noname-b con 0x7f23140a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.081+0000 7f231ba6e640 10 monclient(hunting): picked mon.noname-c con 0x7f231413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.081+0000 7f231ba6e640 10 monclient(hunting): picked mon.noname-a con 0x7f23141430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.081+0000 7f231ba6e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2314142520 con 0x7f23141430c0 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.081+0000 7f231ba6e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140b28a0 con 0x7f231413f710 2024-03-25T19:44:47.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.081+0000 7f231ba6e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140b2a30 con 0x7f23140a98f0 2024-03-25T19:44:47.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.081+0000 7f231ba6e640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:47.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.081+0000 7f231ba6e640 10 monclient(hunting): authenticate will time out at 4458.223125s 2024-03-25T19:44:47.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.086+0000 7f2319009640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f231413f710 0x7f23141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:44658/0 (socket says 172.21.15.110:44658) 2024-03-25T19:44:47.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.086+0000 7f2319009640 1 -- 172.21.15.110:0/1044442543 learned_addr learned my addr 172.21.15.110:0/1044442543 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:47.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.100+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481020682 0 0) 0x7f23140b28a0 con 0x7f231413f710 2024-03-25T19:44:47.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.100+0000 7f2318808640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.100+0000 7f2318808640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.100+0000 7f2318808640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.100+0000 7f2318808640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:44:47.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.100+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2300003520 con 0x7f231413f710 2024-03-25T19:44:47.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.102+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814442471 0 0) 0x7f2314142520 con 0x7f23141430c0 2024-03-25T19:44:47.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.102+0000 7f2318808640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.102+0000 7f2318808640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.102+0000 7f2318808640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.102+0000 7f2318808640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:44:47.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.102+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23140b28a0 con 0x7f23141430c0 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 582829111 0 0) 0x7f23140b28a0 con 0x7f23141430c0 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 >> v1:172.21.15.110:6790/0 conn(0x7f23140a98f0 legacy=0x7f23140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 >> v1:172.21.15.178:6789/0 conn(0x7f231413f710 legacy=0x7f23141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 10 monclient: _finish_hunting 0 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 1 monclient: found mon.noname-a 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140b44c0 con 0x7f23141430c0 2024-03-25T19:44:47.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 10 monclient: _finish_auth 0 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 10 monclient: _check_auth_tickets 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f230c0019f0 con 0x7f23141430c0 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f231ba6e640 5 monclient: authenticate success, global_id 4410 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f231ba6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f2318808640 10 monclient: dump: 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.117 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:47.118 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:47.118 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:47.118 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:47.118 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:47.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.115+0000 7f231ba6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:47.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f230c001b50 con 0x7f23141430c0 2024-03-25T19:44:47.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f2318808640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f2318808640 1 -- 172.21.15.110:0/1044442543 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f230c001f20 con 0x7f23141430c0 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f231ba6e640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f231ba6e640 4 set_mon_vals no callback set 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f2318808640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f2318808640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f2318808640 10 monclient: dump: 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:47.127 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f231ba6e640 10 monclient: shutdown 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f231ba6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f231ba6e640 1 -- 172.21.15.110:0/1044442543 >> v1:172.21.15.110:6789/0 conn(0x7f23141430c0 legacy=0x7f231414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f231ba6e640 1 -- 172.21.15.110:0/1044442543 shutdown_connections 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.126+0000 7f231ba6e640 1 -- 172.21.15.110:0/1044442543 >> 172.21.15.110:0/1044442543 conn(0x7f23140b7ea0 msgr2=0x7f23140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.127+0000 7f231ba6e640 1 -- 172.21.15.110:0/1044442543 shutdown_connections 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.127+0000 7f231ba6e640 1 -- 172.21.15.110:0/1044442543 wait complete. 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.127+0000 7f231ba6e640 10 monclient: build_initial_monmap 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.127+0000 7f231ba6e640 10 monclient: monmap: 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:47.130 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 librados: starting msgr at 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 librados: starting objecter 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 Processor -- start 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 -- start start 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 librados: setting wanted keys 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 librados: calling monclient init 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 10 monclient: init 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 10 monclient(hunting): picked mon.noname-a con 0x7f23140a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 10 monclient(hunting): picked mon.noname-b con 0x7f231413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:47.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 10 monclient(hunting): picked mon.noname-c con 0x7f23141430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:47.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140ab950 con 0x7f23140a98f0 2024-03-25T19:44:47.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23141db6f0 con 0x7f231413f710 2024-03-25T19:44:47.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23141db410 con 0x7f23141430c0 2024-03-25T19:44:47.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:47.132 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.128+0000 7f231ba6e640 10 monclient(hunting): authenticate will time out at 4458.270664s 2024-03-25T19:44:47.133 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.133+0000 7f2319009640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f231413f710 0x7f23140b0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:44690/0 (socket says 172.21.15.110:44690) 2024-03-25T19:44:47.133 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.133+0000 7f2319009640 1 -- 172.21.15.110:0/3664689877 learned_addr learned my addr 172.21.15.110:0/3664689877 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:47.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.136+0000 7f230a7fc640 10 client.?.objecter ms_handle_connect 0x7f231413f710 2024-03-25T19:44:47.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.136+0000 7f230a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:47.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.137+0000 7f230a7fc640 10 client.?.objecter ms_handle_connect 0x7f23140a98f0 2024-03-25T19:44:47.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.137+0000 7f230a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:47.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.141+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171980726 0 0) 0x7f23141db6f0 con 0x7f231413f710 2024-03-25T19:44:47.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.141+0000 7f230a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.141+0000 7f230a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.141+0000 7f230a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.141+0000 7f230a7fc640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:44:47.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.141+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f8004eb0 con 0x7f231413f710 2024-03-25T19:44:47.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.148+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862488582 0 0) 0x7f23140ab950 con 0x7f23140a98f0 2024-03-25T19:44:47.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.148+0000 7f230a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.148+0000 7f230a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.148+0000 7f230a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.148+0000 7f230a7fc640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:44:47.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.148+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23141db6f0 con 0x7f23140a98f0 2024-03-25T19:44:47.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.149+0000 7f230a7fc640 10 client.?.objecter ms_handle_connect 0x7f23141430c0 2024-03-25T19:44:47.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.149+0000 7f230a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.152+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3088987648 0 0) 0x7f22f8004eb0 con 0x7f231413f710 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.152+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23140ab950 con 0x7f231413f710 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.152+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2304001950 con 0x7f231413f710 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.152+0000 7f230a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.152+0000 7f230a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.152+0000 7f230a7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:47.153 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:47.154 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:47.154 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:47.154 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:47.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.152+0000 7f230a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:47.161 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.160+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835045670 0 0) 0x7f23141db6f0 con 0x7f23140a98f0 2024-03-25T19:44:47.161 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.160+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f8004eb0 con 0x7f23140a98f0 2024-03-25T19:44:47.161 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.160+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f23100018f0 con 0x7f23140a98f0 2024-03-25T19:44:47.161 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.160+0000 7f230a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.161 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.160+0000 7f230a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:47.161 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.160+0000 7f230a7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:47.161 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:47.162 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:47.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.167+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304510831 0 0) 0x7f23141db410 con 0x7f23141430c0 2024-03-25T19:44:47.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.167+0000 7f230a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.167+0000 7f230a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.167+0000 7f230a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.167+0000 7f230a7fc640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:44:47.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.167+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23141db6f0 con 0x7f23141430c0 2024-03-25T19:44:47.170 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1863064984 0 0) 0x7f23140ab950 con 0x7f231413f710 2024-03-25T19:44:47.170 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 >> v1:172.21.15.110:6790/0 conn(0x7f23141430c0 legacy=0x7f23141d4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 >> v1:172.21.15.110:6789/0 conn(0x7f23140a98f0 legacy=0x7f23141dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 1 monclient: found mon.b 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140b1360 con 0x7f231413f710 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f230a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 10 monclient: _renew_subs 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23140b15b0 con 0x7f231413f710 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 10 monclient: _renew_subs 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:47.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23140b0af0 con 0x7f231413f710 2024-03-25T19:44:47.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.169+0000 7f231ba6e640 1 librados: init done 2024-03-25T19:44:47.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.170+0000 7f231ba6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:47.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.170+0000 7f231ba6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:47.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.170+0000 7f231ba6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:47.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.170+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23141db880 con 0x7f231413f710 2024-03-25T19:44:47.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.454+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23040047e0 con 0x7f231413f710 2024-03-25T19:44:47.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.454+0000 7f230a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:47.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.454+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2304004af0 con 0x7f231413f710 2024-03-25T19:44:47.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.454+0000 7f230a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.454+0000 7f230a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:47.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.454+0000 7f230a7fc640 10 monclient: dump: 2024-03-25T19:44:47.455 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:47.456 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.454+0000 7f230bfff640 4 set_mon_vals no callback set 2024-03-25T19:44:47.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.455+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f230401aac0 con 0x7f231413f710 2024-03-25T19:44:47.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.456+0000 7f231980a640 10 monclient: get_auth_request con 0x7f22f806df00 auth_method 0 2024-03-25T19:44:47.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.456+0000 7f230a7fc640 10 client.4426.objecter ms_handle_connect 0x7f22f806df00 2024-03-25T19:44:47.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.459+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f2304054850 con 0x7f231413f710 2024-03-25T19:44:47.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.459+0000 7f230a7fc640 10 client.4426.objecter ms_dispatch 0x7f2314000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:47.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.459+0000 7f230a7fc640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:47.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.459+0000 7f230a7fc640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:47.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.460+0000 7f230a7fc640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T19:44:47.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.460+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2304050360 con 0x7f231413f710 2024-03-25T19:44:47.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.460+0000 7f230a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:47.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.460+0000 7f230a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:47.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.705+0000 7f231ba6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.705+0000 7f231ba6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.705+0000 7f231ba6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:47.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.705+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23140627b0 con 0x7f231413f710 2024-03-25T19:44:47.737 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.736+0000 7f230a7fc640 1 -- 172.21.15.110:0/3664689877 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f2304052290 con 0x7f231413f710 2024-03-25T19:44:47.737 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.736+0000 7f230a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.737 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.736+0000 7f230a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:47.737 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:44:47.737 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 10 librados: watch_flush enter 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 10 librados: watch_flush exit 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 10 client.4426.objecter successfully canceled tick 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 >> v1:172.21.15.178:6800/1132739058 conn(0x7f22f806df00 legacy=0x7f22f80703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 10 monclient: shutdown 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 >> v1:172.21.15.178:6789/0 conn(0x7f231413f710 legacy=0x7f23140b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 shutdown_connections 2024-03-25T19:44:47.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 >> 172.21.15.110:0/3664689877 conn(0x7f23140b7ea0 msgr2=0x7f2314074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:47.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 shutdown_connections 2024-03-25T19:44:47.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 1 -- 172.21.15.110:0/3664689877 wait complete. 2024-03-25T19:44:47.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:47.739+0000 7f231ba6e640 1 librados: shutdown 2024-03-25T19:44:47.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:54.557 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:54.674 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.675+0000 7f7755f0e640 10 monclient: get_monmap_and_config 2024-03-25T19:44:54.674 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.675+0000 7f7755f0e640 10 monclient: build_initial_monmap 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.675+0000 7f7755f0e640 10 monclient: monmap: 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:44:54.676119+0000 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:44:54.676119+0000 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:54.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.675+0000 7f7755f0e640 1 Processor -- start 2024-03-25T19:44:54.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 1 -- start start 2024-03-25T19:44:54.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 10 monclient: init 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f77500a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f775013f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f77501430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7750142520 con 0x7f77501430c0 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500b28a0 con 0x7f77500a98f0 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500b2a30 con 0x7f775013f710 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:54.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.676+0000 7f7755f0e640 10 monclient(hunting): authenticate will time out at 4465.817518s 2024-03-25T19:44:54.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.681+0000 7f774effd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f775013f710 0x7f77501429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:49180/0 (socket says 172.21.15.110:49180) 2024-03-25T19:44:54.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.681+0000 7f774effd640 1 -- 172.21.15.110:0/856899643 learned_addr learned my addr 172.21.15.110:0/856899643 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.692+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953307238 0 0) 0x7f7750142520 con 0x7f77501430c0 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.692+0000 7f774e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.692+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.692+0000 7f774e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.692+0000 7f774e7fc640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.692+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7738003520 con 0x7f77501430c0 2024-03-25T19:44:54.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950222573 0 0) 0x7f77500b2a30 con 0x7f775013f710 2024-03-25T19:44:54.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:44:54.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7750142520 con 0x7f775013f710 2024-03-25T19:44:54.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3617208688 0 0) 0x7f7738003520 con 0x7f77501430c0 2024-03-25T19:44:54.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 >> v1:172.21.15.110:6790/0 conn(0x7f775013f710 legacy=0x7f77501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 >> v1:172.21.15.178:6789/0 conn(0x7f77500a98f0 legacy=0x7f77500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77500b44c0 con 0x7f77501430c0 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f7748001a10 con 0x7f77501430c0 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f7755f0e640 5 monclient: authenticate success, global_id 4416 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f7755f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f774e7fc640 10 monclient: dump: 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:54.701 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:54.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.700+0000 7f7755f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.709+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7748001b70 con 0x7f77501430c0 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.709+0000 7f774e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.709+0000 7f774e7fc640 1 -- 172.21.15.110:0/856899643 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f7748001f80 con 0x7f77501430c0 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.709+0000 7f7755f0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.709+0000 7f7755f0e640 4 set_mon_vals no callback set 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.709+0000 7f774e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.709+0000 7f774e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.709+0000 7f774e7fc640 10 monclient: dump: 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:54.710 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.710+0000 7f7755f0e640 10 monclient: shutdown 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.710+0000 7f7755f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.710+0000 7f7755f0e640 1 -- 172.21.15.110:0/856899643 >> v1:172.21.15.110:6789/0 conn(0x7f77501430c0 legacy=0x7f775014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.710+0000 7f7755f0e640 1 -- 172.21.15.110:0/856899643 shutdown_connections 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.710+0000 7f7755f0e640 1 -- 172.21.15.110:0/856899643 >> 172.21.15.110:0/856899643 conn(0x7f77500b7ea0 msgr2=0x7f77500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.710+0000 7f7755f0e640 1 -- 172.21.15.110:0/856899643 shutdown_connections 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.710+0000 7f7755f0e640 1 -- 172.21.15.110:0/856899643 wait complete. 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.711+0000 7f7755f0e640 10 monclient: build_initial_monmap 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.711+0000 7f7755f0e640 10 monclient: monmap: 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:44:54.712 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.711+0000 7f7755f0e640 1 librados: starting msgr at 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.711+0000 7f7755f0e640 1 librados: starting objecter 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.711+0000 7f7755f0e640 1 Processor -- start 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 1 -- start start 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 1 librados: setting wanted keys 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 1 librados: calling monclient init 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 10 monclient: init 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:54.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f77500a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f775013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f77501430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7750136a50 con 0x7f775013f710 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500b79c0 con 0x7f77500a98f0 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500b7ca0 con 0x7f77501430c0 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.712+0000 7f7755f0e640 10 monclient(hunting): authenticate will time out at 4465.853420s 2024-03-25T19:44:54.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.713+0000 7f774f7fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f77500a98f0 0x7f7750137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:47322/0 (socket says 172.21.15.110:47322) 2024-03-25T19:44:54.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.713+0000 7f774f7fe640 1 -- 172.21.15.110:0/80264459 learned_addr learned my addr 172.21.15.110:0/80264459 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:44:54.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.713+0000 7f7737fff640 10 client.?.objecter ms_handle_connect 0x7f77500a98f0 2024-03-25T19:44:54.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.713+0000 7f7737fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.719 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.719+0000 7f7737fff640 10 client.?.objecter ms_handle_connect 0x7f77501430c0 2024-03-25T19:44:54.719 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.719+0000 7f7737fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.720+0000 7f7737fff640 10 client.?.objecter ms_handle_connect 0x7f775013f710 2024-03-25T19:44:54.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.720+0000 7f7737fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.730+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004207198 0 0) 0x7f7750136a50 con 0x7f775013f710 2024-03-25T19:44:54.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.730+0000 7f7737fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.730+0000 7f7737fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.730+0000 7f7737fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.730+0000 7f7737fff640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:44:54.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.730+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f773c004eb0 con 0x7f775013f710 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.801+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338877120 0 0) 0x7f77500b79c0 con 0x7f77500a98f0 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.801+0000 7f7737fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.801+0000 7f7737fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.801+0000 7f7737fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.801+0000 7f7737fff640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.801+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7750136a50 con 0x7f77500a98f0 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.807+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3408651124 0 0) 0x7f7750136a50 con 0x7f77500a98f0 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.807+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77500b79c0 con 0x7f77500a98f0 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.807+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f7740001970 con 0x7f77500a98f0 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.807+0000 7f7737fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.807+0000 7f7737fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.807+0000 7f7737fff640 10 monclient(hunting): dump: 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:54.808 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:54.808 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:54.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.807+0000 7f7737fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:54.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.822+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3791691783 0 0) 0x7f773c004eb0 con 0x7f775013f710 2024-03-25T19:44:54.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.823+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7750136a50 con 0x7f775013f710 2024-03-25T19:44:54.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.823+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f7744001930 con 0x7f775013f710 2024-03-25T19:44:54.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.823+0000 7f7737fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.823+0000 7f7737fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:54.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.823+0000 7f7737fff640 10 monclient(hunting): dump: 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:54.823 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:54.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 893779651 0 0) 0x7f77500b79c0 con 0x7f77500a98f0 2024-03-25T19:44:54.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 >> v1:172.21.15.110:6790/0 conn(0x7f77501430c0 legacy=0x7f77501e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 >> v1:172.21.15.110:6789/0 conn(0x7f775013f710 legacy=0x7f7750135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 1 monclient: found mon.b 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750136ca0 con 0x7f77500a98f0 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7737fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7755f0e640 5 monclient: authenticate success, global_id 4429 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7755f0e640 10 monclient: _renew_subs 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7755f0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.832+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7750136f80 con 0x7f77500a98f0 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 10 monclient: _renew_subs 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:54.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77501360e0 con 0x7f77500a98f0 2024-03-25T19:44:54.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 1 librados: init done 2024-03-25T19:44:54.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:54.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:54.833+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77500627b0 con 0x7f77500a98f0 2024-03-25T19:44:55.107 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.107+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7740003610 con 0x7f77500a98f0 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.108+0000 7f7737fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.108+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f77400039a0 con 0x7f77500a98f0 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.108+0000 7f7737fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.108+0000 7f7737fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.108+0000 7f7737fff640 10 monclient: dump: 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:44:55.108 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:44:55.109 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:44:55.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.108+0000 7f774dffb640 4 set_mon_vals no callback set 2024-03-25T19:44:55.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.108+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f7740003b00 con 0x7f77500a98f0 2024-03-25T19:44:55.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.109+0000 7f774effd640 10 monclient: get_auth_request con 0x7f773c06de40 auth_method 0 2024-03-25T19:44:55.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.109+0000 7f7737fff640 10 client.4429.objecter ms_handle_connect 0x7f773c06de40 2024-03-25T19:44:55.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.115+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f77400547c0 con 0x7f77500a98f0 2024-03-25T19:44:55.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.115+0000 7f7737fff640 10 client.4429.objecter ms_dispatch 0x7f7750000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:55.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.115+0000 7f7737fff640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:55.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.115+0000 7f7737fff640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:55.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.115+0000 7f7737fff640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T19:44:55.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.115+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77400502d0 con 0x7f77500a98f0 2024-03-25T19:44:55.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.115+0000 7f7737fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:55.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.115+0000 7f7737fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:55.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.367+0000 7f7755f0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.367+0000 7f7755f0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.367+0000 7f7755f0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:44:55.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.367+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77500008d0 con 0x7f77500a98f0 2024-03-25T19:44:55.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.400+0000 7f7737fff640 1 -- 172.21.15.110:0/80264459 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f7740052200 con 0x7f77500a98f0 2024-03-25T19:44:55.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.400+0000 7f7737fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.400+0000 7f7737fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:55.400 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:44:55.400 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 10 librados: watch_flush enter 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 10 librados: watch_flush exit 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 10 client.4429.objecter successfully canceled tick 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 >> v1:172.21.15.178:6800/1132739058 conn(0x7f773c06de40 legacy=0x7f773c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 10 monclient: shutdown 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 >> v1:172.21.15.178:6789/0 conn(0x7f77500a98f0 legacy=0x7f7750137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 shutdown_connections 2024-03-25T19:44:55.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 >> 172.21.15.110:0/80264459 conn(0x7f77500b7ea0 msgr2=0x7f7750074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:55.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 shutdown_connections 2024-03-25T19:44:55.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 1 -- 172.21.15.110:0/80264459 wait complete. 2024-03-25T19:44:55.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:44:55.403+0000 7f7755f0e640 1 librados: shutdown 2024-03-25T19:44:55.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:57.607 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:04.891 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.657 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:04.941 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.679 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:04.964 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.711 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:04.965 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.715 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:04.999 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.716 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:04.970 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.725 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:05.010 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.753 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:05.007 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.762 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:05.047 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.221 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:02.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.339+0000 7f231c4c8640 10 monclient: get_monmap_and_config 2024-03-25T19:45:02.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.339+0000 7f231c4c8640 10 monclient: build_initial_monmap 2024-03-25T19:45:02.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.339+0000 7f231c4c8640 10 monclient: monmap: 2024-03-25T19:45:02.339 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:02.339 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:02.340 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:45:02.340716+0000 2024-03-25T19:45:02.340 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:45:02.340716+0000 2024-03-25T19:45:02.340 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:02.340 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:02.340 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:02.340 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:45:02.340 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:45:02.340 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:02.341 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.340+0000 7f231c4c8640 1 Processor -- start 2024-03-25T19:45:02.341 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.340+0000 7f231c4c8640 1 -- start start 2024-03-25T19:45:02.341 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.340+0000 7f231c4c8640 10 monclient: init 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 10 monclient(hunting): picked mon.noname-c con 0x7f23140a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 10 monclient(hunting): picked mon.noname-a con 0x7f231413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 10 monclient(hunting): picked mon.noname-b con 0x7f23141430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2314142520 con 0x7f231413f710 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140b28a0 con 0x7f23140a98f0 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140b2a30 con 0x7f23141430c0 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.341+0000 7f231c4c8640 10 monclient(hunting): authenticate will time out at 4473.482577s 2024-03-25T19:45:02.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.349+0000 7f2319a63640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f231413f710 0x7f23141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55226/0 (socket says 172.21.15.110:55226) 2024-03-25T19:45:02.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.349+0000 7f2319a63640 1 -- 172.21.15.110:0/514253444 learned_addr learned my addr 172.21.15.110:0/514253444 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:02.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.359+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446908061 0 0) 0x7f2314142520 con 0x7f231413f710 2024-03-25T19:45:02.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.359+0000 7f2319262640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.361 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.359+0000 7f2319262640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.361 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.359+0000 7f2319262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.361 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.359+0000 7f2319262640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:45:02.361 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.359+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2300003520 con 0x7f231413f710 2024-03-25T19:45:02.362 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.361+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847420184 0 0) 0x7f23140b2a30 con 0x7f23141430c0 2024-03-25T19:45:02.362 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.361+0000 7f2319262640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.362 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.361+0000 7f2319262640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.362 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.361+0000 7f2319262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.362 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.361+0000 7f2319262640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:45:02.362 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.361+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2314142520 con 0x7f23141430c0 2024-03-25T19:45:02.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.371+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113288719 0 0) 0x7f23140b28a0 con 0x7f23140a98f0 2024-03-25T19:45:02.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.371+0000 7f2319262640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.371+0000 7f2319262640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.371+0000 7f2319262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.371+0000 7f2319262640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:45:02.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.371+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23140b2a30 con 0x7f23140a98f0 2024-03-25T19:45:02.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2273436736 0 0) 0x7f2300003520 con 0x7f231413f710 2024-03-25T19:45:02.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 >> v1:172.21.15.110:6790/0 conn(0x7f23141430c0 legacy=0x7f231414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 >> v1:172.21.15.178:6789/0 conn(0x7f23140a98f0 legacy=0x7f23140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 10 monclient: _finish_hunting 0 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 1 monclient: found mon.noname-a 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140b44c0 con 0x7f231413f710 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 10 monclient: _finish_auth 0 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 10 monclient: _check_auth_tickets 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2304001690 con 0x7f231413f710 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f231c4c8640 5 monclient: authenticate success, global_id 4422 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f231c4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:02.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f2319262640 10 monclient: dump: 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:02.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.375+0000 7f231c4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.384+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23040017f0 con 0x7f231413f710 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.384+0000 7f2319262640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.384+0000 7f2319262640 1 -- 172.21.15.110:0/514253444 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2304001c00 con 0x7f231413f710 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.384+0000 7f231c4c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.384+0000 7f231c4c8640 4 set_mon_vals no callback set 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.384+0000 7f2319262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.384+0000 7f2319262640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.384+0000 7f2319262640 10 monclient: dump: 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:02.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.385+0000 7f231c4c8640 10 monclient: shutdown 2024-03-25T19:45:02.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.385+0000 7f231c4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:02.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.385+0000 7f231c4c8640 1 -- 172.21.15.110:0/514253444 >> v1:172.21.15.110:6789/0 conn(0x7f231413f710 legacy=0x7f23141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.385+0000 7f231c4c8640 1 -- 172.21.15.110:0/514253444 shutdown_connections 2024-03-25T19:45:02.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.385+0000 7f231c4c8640 1 -- 172.21.15.110:0/514253444 >> 172.21.15.110:0/514253444 conn(0x7f23140b7ea0 msgr2=0x7f23140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:02.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.385+0000 7f231c4c8640 1 -- 172.21.15.110:0/514253444 shutdown_connections 2024-03-25T19:45:02.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.385+0000 7f231c4c8640 1 -- 172.21.15.110:0/514253444 wait complete. 2024-03-25T19:45:02.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 10 monclient: build_initial_monmap 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 10 monclient: monmap: 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 1 librados: starting msgr at 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 1 librados: starting objecter 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 1 Processor -- start 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 1 -- start start 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 1 librados: setting wanted keys 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 1 librados: calling monclient init 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.386+0000 7f231c4c8640 10 monclient: init 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 10 monclient(hunting): picked mon.noname-b con 0x7f23140a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 10 monclient(hunting): picked mon.noname-a con 0x7f231413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 10 monclient(hunting): picked mon.noname-c con 0x7f23141430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2314139520 con 0x7f231413f710 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2314138b40 con 0x7f23140a98f0 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2314138e20 con 0x7f23141430c0 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.387+0000 7f231c4c8640 10 monclient(hunting): authenticate will time out at 4473.528135s 2024-03-25T19:45:02.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.396+0000 7f231a264640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f23140a98f0 0x7f2314137460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:47500/0 (socket says 172.21.15.110:47500) 2024-03-25T19:45:02.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.396+0000 7f231a264640 1 -- 172.21.15.110:0/438706200 learned_addr learned my addr 172.21.15.110:0/438706200 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:02.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.397+0000 7f230affd640 10 client.?.objecter ms_handle_connect 0x7f231413f710 2024-03-25T19:45:02.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.398+0000 7f230affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.399+0000 7f230affd640 10 client.?.objecter ms_handle_connect 0x7f23140a98f0 2024-03-25T19:45:02.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.399+0000 7f230affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.406+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912295365 0 0) 0x7f2314139520 con 0x7f231413f710 2024-03-25T19:45:02.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.406+0000 7f230affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.406+0000 7f230affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.406+0000 7f230affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.406+0000 7f230affd640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:45:02.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.406+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f8004eb0 con 0x7f231413f710 2024-03-25T19:45:02.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.409+0000 7f230affd640 10 client.?.objecter ms_handle_connect 0x7f23141430c0 2024-03-25T19:45:02.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.409+0000 7f230affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.413+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270454348 0 0) 0x7f2314138b40 con 0x7f23140a98f0 2024-03-25T19:45:02.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.413+0000 7f230affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.413+0000 7f230affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.413+0000 7f230affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.413+0000 7f230affd640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:45:02.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.413+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2314139520 con 0x7f23140a98f0 2024-03-25T19:45:02.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.419+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989930327 0 0) 0x7f2314138e20 con 0x7f23141430c0 2024-03-25T19:45:02.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.419+0000 7f230affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.419+0000 7f230affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.419+0000 7f230affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.419+0000 7f230affd640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:45:02.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.419+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2314138b40 con 0x7f23141430c0 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.420+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3298631375 0 0) 0x7f22f8004eb0 con 0x7f231413f710 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.420+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2314138e20 con 0x7f231413f710 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.420+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2304001b00 con 0x7f231413f710 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.420+0000 7f230affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.420+0000 7f230affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.420+0000 7f230affd640 10 monclient(hunting): dump: 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:02.420 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:02.421 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:02.421 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.421 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:02.421 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:02.421 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:02.421 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:02.421 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:02.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.420+0000 7f230affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4235393842 0 0) 0x7f2314138e20 con 0x7f231413f710 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 >> v1:172.21.15.110:6790/0 conn(0x7f23141430c0 legacy=0x7f23141e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 >> v1:172.21.15.178:6789/0 conn(0x7f23140a98f0 legacy=0x7f2314137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 1 monclient: found mon.a 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2314139100 con 0x7f231413f710 2024-03-25T19:45:02.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 10 monclient: _finish_auth 0 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f230affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 5 monclient: authenticate success, global_id 4425 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 10 monclient: _renew_subs 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2314136aa0 con 0x7f231413f710 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 10 monclient: _renew_subs 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f231413ba10 con 0x7f231413f710 2024-03-25T19:45:02.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.432+0000 7f231c4c8640 1 librados: init done 2024-03-25T19:45:02.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.433+0000 7f231c4c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.433+0000 7f231c4c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.433+0000 7f231c4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:02.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.433+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23140627b0 con 0x7f231413f710 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.746+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2304001220 con 0x7f231413f710 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.746+0000 7f230affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.746+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2304004c30 con 0x7f231413f710 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.746+0000 7f230affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.746+0000 7f230affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.746+0000 7f230affd640 10 monclient: dump: 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:02.746 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:02.747 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:02.747 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.747 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:02.747 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:02.747 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:02.747 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:02.747 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:02.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.746+0000 7f2318a61640 4 set_mon_vals no callback set 2024-03-25T19:45:02.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.747+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f230401ac00 con 0x7f231413f710 2024-03-25T19:45:02.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.748+0000 7f231a264640 10 monclient: get_auth_request con 0x7f22f806dda0 auth_method 0 2024-03-25T19:45:02.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.748+0000 7f230affd640 10 client.4425.objecter ms_handle_connect 0x7f22f806dda0 2024-03-25T19:45:02.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.754+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f2304054860 con 0x7f231413f710 2024-03-25T19:45:02.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.754+0000 7f230affd640 10 client.4425.objecter ms_dispatch 0x7f2314000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:02.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.754+0000 7f230affd640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:02.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.754+0000 7f230affd640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:02.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.754+0000 7f230affd640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-25T19:45:02.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.754+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23040501a0 con 0x7f231413f710 2024-03-25T19:45:02.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.754+0000 7f230affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:02.754+0000 7f230affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:03.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.018+0000 7f231c4c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:03.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.018+0000 7f231c4c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:03.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.018+0000 7f231c4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:03.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.018+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23141398c0 con 0x7f231413f710 2024-03-25T19:45:03.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.045+0000 7f230affd640 1 -- 172.21.15.110:0/438706200 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f23040522a0 con 0x7f231413f710 2024-03-25T19:45:03.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.045+0000 7f230affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:03.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.045+0000 7f230affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:03.045 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:45:03.045 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.047+0000 7f231c4c8640 10 librados: watch_flush enter 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.047+0000 7f231c4c8640 10 librados: watch_flush exit 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.047+0000 7f231c4c8640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.047+0000 7f231c4c8640 10 client.4425.objecter successfully canceled tick 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 >> v1:172.21.15.178:6800/1132739058 conn(0x7f22f806dda0 legacy=0x7f22f8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 10 monclient: shutdown 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 >> v1:172.21.15.110:6789/0 conn(0x7f231413f710 legacy=0x7f2314137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 shutdown_connections 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 >> 172.21.15.110:0/438706200 conn(0x7f23140b7ea0 msgr2=0x7f2314074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 shutdown_connections 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 1 -- 172.21.15.110:0/438706200 wait complete. 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:03.048+0000 7f231c4c8640 1 librados: shutdown 2024-03-25T19:45:03.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:09.867 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.978+0000 7f98e0c55640 10 monclient: get_monmap_and_config 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.978+0000 7f98e0c55640 10 monclient: build_initial_monmap 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.978+0000 7f98e0c55640 10 monclient: monmap: 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:45:09.979377+0000 2024-03-25T19:45:09.979 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:45:09.979377+0000 2024-03-25T19:45:09.979 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:09.979 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:09.979 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:09.979 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:45:09.979 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:45:09.979 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:09.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.979+0000 7f98e0c55640 1 Processor -- start 2024-03-25T19:45:09.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.979+0000 7f98e0c55640 1 -- start start 2024-03-25T19:45:09.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.979+0000 7f98e0c55640 10 monclient: init 2024-03-25T19:45:09.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:09.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:09.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 10 monclient(hunting): picked mon.noname-c con 0x7f98dc0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:09.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 10 monclient(hunting): picked mon.noname-b con 0x7f98dc13f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:09.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 10 monclient(hunting): picked mon.noname-a con 0x7f98dc1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:09.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc142520 con 0x7f98dc1430c0 2024-03-25T19:45:09.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0b28a0 con 0x7f98dc0a98f0 2024-03-25T19:45:09.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0b2a30 con 0x7f98dc13f710 2024-03-25T19:45:09.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:09.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:09.980+0000 7f98e0c55640 10 monclient(hunting): authenticate will time out at 4481.121228s 2024-03-25T19:45:10.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.003+0000 7f98da59c640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f98dc0a98f0 0x7f98dc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:52216/0 (socket says 172.21.15.110:52216) 2024-03-25T19:45:10.003 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.003+0000 7f98da59c640 1 -- 172.21.15.110:0/3995362461 learned_addr learned my addr 172.21.15.110:0/3995362461 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.010+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790525250 0 0) 0x7f98dc0b28a0 con 0x7f98dc0a98f0 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.010+0000 7f98d959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.010+0000 7f98d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.010+0000 7f98d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.010+0000 7f98d959a640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.010+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c4003520 con 0x7f98dc0a98f0 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.012+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477414218 0 0) 0x7f98dc0b2a30 con 0x7f98dc13f710 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.012+0000 7f98d959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.012+0000 7f98d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.012+0000 7f98d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.012+0000 7f98d959a640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:45:10.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.012+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98dc0b28a0 con 0x7f98dc13f710 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3384477226 0 0) 0x7f98dc0b28a0 con 0x7f98dc13f710 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 >> v1:172.21.15.178:6789/0 conn(0x7f98dc0a98f0 legacy=0x7f98dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 >> v1:172.21.15.110:6789/0 conn(0x7f98dc1430c0 legacy=0x7f98dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 10 monclient: _finish_hunting 0 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 1 monclient: found mon.noname-b 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0b44c0 con 0x7f98dc13f710 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 10 monclient: _finish_auth 0 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 10 monclient: _check_auth_tickets 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:10.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f98d0001670 con 0x7f98dc13f710 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98e0c55640 5 monclient: authenticate success, global_id 4436 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98e0c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98d959a640 10 monclient: dump: 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:10.025 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:10.026 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:10.026 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:10.026 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:10.026 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.024+0000 7f98e0c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.031+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d00017d0 con 0x7f98dc13f710 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.031+0000 7f98d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.031+0000 7f98d959a640 1 -- 172.21.15.110:0/3995362461 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f98d0001ba0 con 0x7f98dc13f710 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.031+0000 7f98e0c55640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.031+0000 7f98e0c55640 4 set_mon_vals no callback set 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.031+0000 7f98d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.031+0000 7f98d959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.031+0000 7f98d959a640 10 monclient: dump: 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:10.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.032+0000 7f98e0c55640 10 monclient: shutdown 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.032+0000 7f98e0c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.032+0000 7f98e0c55640 1 -- 172.21.15.110:0/3995362461 >> v1:172.21.15.110:6790/0 conn(0x7f98dc13f710 legacy=0x7f98dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.032+0000 7f98e0c55640 1 -- 172.21.15.110:0/3995362461 shutdown_connections 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.032+0000 7f98e0c55640 1 -- 172.21.15.110:0/3995362461 >> 172.21.15.110:0/3995362461 conn(0x7f98dc0b7ea0 msgr2=0x7f98dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.032+0000 7f98e0c55640 1 -- 172.21.15.110:0/3995362461 shutdown_connections 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.032+0000 7f98e0c55640 1 -- 172.21.15.110:0/3995362461 wait complete. 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.033+0000 7f98e0c55640 10 monclient: build_initial_monmap 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.033+0000 7f98e0c55640 10 monclient: monmap: 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:45:10.035 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.033+0000 7f98e0c55640 1 librados: starting msgr at 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.033+0000 7f98e0c55640 1 librados: starting objecter 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.033+0000 7f98e0c55640 1 Processor -- start 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 1 -- start start 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 1 librados: setting wanted keys 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 1 librados: calling monclient init 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 10 monclient: init 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 10 monclient(hunting): picked mon.noname-c con 0x7f98dc0a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:10.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 10 monclient(hunting): picked mon.noname-b con 0x7f98dc13f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:10.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 10 monclient(hunting): picked mon.noname-a con 0x7f98dc1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:10.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc13b720 con 0x7f98dc1430c0 2024-03-25T19:45:10.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc139d20 con 0x7f98dc13f710 2024-03-25T19:45:10.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0b7b70 con 0x7f98dc0a98f0 2024-03-25T19:45:10.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:10.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.034+0000 7f98e0c55640 10 monclient(hunting): authenticate will time out at 4481.175470s 2024-03-25T19:45:10.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.040+0000 7f98dad9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f98dc1430c0 0x7f98dc1e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59850/0 (socket says 172.21.15.110:59850) 2024-03-25T19:45:10.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.040+0000 7f98dad9d640 1 -- 172.21.15.110:0/1304434372 learned_addr learned my addr 172.21.15.110:0/1304434372 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:10.042 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.042+0000 7f98caffd640 10 client.?.objecter ms_handle_connect 0x7f98dc1430c0 2024-03-25T19:45:10.042 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.042+0000 7f98caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.044+0000 7f98caffd640 10 client.?.objecter ms_handle_connect 0x7f98dc13f710 2024-03-25T19:45:10.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.044+0000 7f98caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081093133 0 0) 0x7f98dc139d20 con 0x7f98dc13f710 2024-03-25T19:45:10.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:45:10.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc004eb0 con 0x7f98dc13f710 2024-03-25T19:45:10.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150275389 0 0) 0x7f98dc13b720 con 0x7f98dc1430c0 2024-03-25T19:45:10.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:45:10.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.054+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98dc139d20 con 0x7f98dc1430c0 2024-03-25T19:45:10.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.058+0000 7f98caffd640 10 client.?.objecter ms_handle_connect 0x7f98dc0a98f0 2024-03-25T19:45:10.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.058+0000 7f98caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.069+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452623087 0 0) 0x7f98dc0b7b70 con 0x7f98dc0a98f0 2024-03-25T19:45:10.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.069+0000 7f98caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.069+0000 7f98caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.069+0000 7f98caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.069+0000 7f98caffd640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:45:10.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.069+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98dc13b720 con 0x7f98dc0a98f0 2024-03-25T19:45:10.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.071+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1294262452 0 0) 0x7f98dc139d20 con 0x7f98dc1430c0 2024-03-25T19:45:10.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.071+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98dc0b7b70 con 0x7f98dc1430c0 2024-03-25T19:45:10.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.071+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f98d4001bf0 con 0x7f98dc1430c0 2024-03-25T19:45:10.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.071+0000 7f98caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.071+0000 7f98caffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:10.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.071+0000 7f98caffd640 10 monclient(hunting): dump: 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:10.072 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.071+0000 7f98caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1369002371 0 0) 0x7f98dc0b7b70 con 0x7f98dc1430c0 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 >> v1:172.21.15.110:6790/0 conn(0x7f98dc0a98f0 legacy=0x7f98dc13bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 >> v1:172.21.15.178:6789/0 conn(0x7f98dc13f710 legacy=0x7f98dc1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 1 monclient: found mon.a 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc13c250 con 0x7f98dc1430c0 2024-03-25T19:45:10.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 5 monclient: authenticate success, global_id 4431 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 10 monclient: _renew_subs 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc0b71f0 con 0x7f98dc1430c0 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 10 monclient: _renew_subs 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc12f910 con 0x7f98dc1430c0 2024-03-25T19:45:10.082 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.081+0000 7f98e0c55640 1 librados: init done 2024-03-25T19:45:10.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.082+0000 7f98e0c55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:10.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.082+0000 7f98e0c55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:10.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.082+0000 7f98e0c55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:10.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.082+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc139eb0 con 0x7f98dc1430c0 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.380+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d40020e0 con 0x7f98dc1430c0 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.380+0000 7f98caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.380+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f98d4003b10 con 0x7f98dc1430c0 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.380+0000 7f98caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.380+0000 7f98caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.380+0000 7f98caffd640 10 monclient: dump: 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:10.380 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.380+0000 7f98d8d99640 4 set_mon_vals no callback set 2024-03-25T19:45:10.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.381+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f98d4003d30 con 0x7f98dc1430c0 2024-03-25T19:45:10.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.381+0000 7f98da59c640 10 monclient: get_auth_request con 0x7f98bc06de20 auth_method 0 2024-03-25T19:45:10.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.381+0000 7f98caffd640 10 client.4431.objecter ms_handle_connect 0x7f98bc06de20 2024-03-25T19:45:10.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.387+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f98d4054a20 con 0x7f98dc1430c0 2024-03-25T19:45:10.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.387+0000 7f98caffd640 10 client.4431.objecter ms_dispatch 0x7f98dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:10.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.387+0000 7f98caffd640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:10.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.387+0000 7f98caffd640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:10.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.387+0000 7f98caffd640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-25T19:45:10.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.387+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98d4050360 con 0x7f98dc1430c0 2024-03-25T19:45:10.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.387+0000 7f98caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:10.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.387+0000 7f98caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:10.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.653+0000 7f98e0c55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.653 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.653+0000 7f98e0c55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.653 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.653+0000 7f98e0c55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:10.653 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.653+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98dc13c3e0 con 0x7f98dc1430c0 2024-03-25T19:45:10.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.683+0000 7f98caffd640 1 -- 172.21.15.110:0/1304434372 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f98d4052460 con 0x7f98dc1430c0 2024-03-25T19:45:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.683+0000 7f98caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.683+0000 7f98caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:10.683 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:45:10.683 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:10.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 10 librados: watch_flush enter 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 10 librados: watch_flush exit 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 10 client.4431.objecter successfully canceled tick 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 >> v1:172.21.15.178:6800/1132739058 conn(0x7f98bc06de20 legacy=0x7f98bc0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 10 monclient: shutdown 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 >> v1:172.21.15.110:6789/0 conn(0x7f98dc1430c0 legacy=0x7f98dc1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 shutdown_connections 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 >> 172.21.15.110:0/1304434372 conn(0x7f98dc0b7ea0 msgr2=0x7f98dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 shutdown_connections 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 1 -- 172.21.15.110:0/1304434372 wait complete. 2024-03-25T19:45:10.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:10.685+0000 7f98e0c55640 1 librados: shutdown 2024-03-25T19:45:10.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:17.505 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:17.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.620+0000 7fd57aed9640 10 monclient: get_monmap_and_config 2024-03-25T19:45:17.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.620+0000 7fd57aed9640 10 monclient: build_initial_monmap 2024-03-25T19:45:17.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.621+0000 7fd57aed9640 10 monclient: monmap: 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:45:17.621928+0000 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:45:17.621928+0000 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:45:17.621 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:17.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.621+0000 7fd57aed9640 1 Processor -- start 2024-03-25T19:45:17.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.621+0000 7fd57aed9640 1 -- start start 2024-03-25T19:45:17.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.621+0000 7fd57aed9640 10 monclient: init 2024-03-25T19:45:17.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 10 monclient(hunting): picked mon.noname-a con 0x7fd5740a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 10 monclient(hunting): picked mon.noname-b con 0x7fd57413f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 10 monclient(hunting): picked mon.noname-c con 0x7fd5741430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd574142520 con 0x7fd5740a98f0 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5740b28a0 con 0x7fd5741430c0 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5740b2a30 con 0x7fd57413f710 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.622+0000 7fd57aed9640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:17.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.623+0000 7fd57aed9640 10 monclient(hunting): authenticate will time out at 4488.763837s 2024-03-25T19:45:17.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.626+0000 7fd579476640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fd5741430c0 0x7fd57414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:50116/0 (socket says 172.21.15.110:50116) 2024-03-25T19:45:17.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.626+0000 7fd579476640 1 -- 172.21.15.110:0/3245440569 learned_addr learned my addr 172.21.15.110:0/3245440569 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:17.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.720+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123074740 0 0) 0x7fd574142520 con 0x7fd5740a98f0 2024-03-25T19:45:17.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.720+0000 7fd5737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.720+0000 7fd5737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.720+0000 7fd5737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.720+0000 7fd5737fe640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:45:17.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.720+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd564003520 con 0x7fd5740a98f0 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3774862622 0 0) 0x7fd564003520 con 0x7fd5740a98f0 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 >> v1:172.21.15.110:6790/0 conn(0x7fd57413f710 legacy=0x7fd5741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 >> v1:172.21.15.178:6789/0 conn(0x7fd5741430c0 legacy=0x7fd57414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 1 monclient: found mon.noname-a 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740b44c0 con 0x7fd5740a98f0 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:17.751 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd568001690 con 0x7fd5740a98f0 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd5737fe640 10 monclient: dump: 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:17.752 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:17.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd57aed9640 5 monclient: authenticate success, global_id 4434 2024-03-25T19:45:17.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.751+0000 7fd57aed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:17.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5680017f0 con 0x7fd5740a98f0 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd5737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd5737fe640 1 -- 172.21.15.110:0/3245440569 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd568001c00 con 0x7fd5740a98f0 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd57aed9640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd57aed9640 4 set_mon_vals no callback set 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd5737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd5737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd5737fe640 10 monclient: dump: 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:17.800 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:17.801 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.800+0000 7fd57aed9640 10 monclient: shutdown 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.801+0000 7fd57aed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.801+0000 7fd57aed9640 1 -- 172.21.15.110:0/3245440569 >> v1:172.21.15.110:6789/0 conn(0x7fd5740a98f0 legacy=0x7fd5740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.801+0000 7fd57aed9640 1 -- 172.21.15.110:0/3245440569 shutdown_connections 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.801+0000 7fd57aed9640 1 -- 172.21.15.110:0/3245440569 >> 172.21.15.110:0/3245440569 conn(0x7fd5740b7ea0 msgr2=0x7fd5740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.801+0000 7fd57aed9640 1 -- 172.21.15.110:0/3245440569 shutdown_connections 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.801+0000 7fd57aed9640 1 -- 172.21.15.110:0/3245440569 wait complete. 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 10 monclient: build_initial_monmap 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 10 monclient: monmap: 2024-03-25T19:45:17.802 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 1 librados: starting msgr at 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 1 librados: starting objecter 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 1 Processor -- start 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 1 -- start start 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 1 librados: setting wanted keys 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 1 librados: calling monclient init 2024-03-25T19:45:17.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.802+0000 7fd57aed9640 10 monclient: init 2024-03-25T19:45:17.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:17.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:17.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 10 monclient(hunting): picked mon.noname-b con 0x7fd57413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:17.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 10 monclient(hunting): picked mon.noname-a con 0x7fd5741430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:17.805 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 10 monclient(hunting): picked mon.noname-c con 0x7fd5740ae7a0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:17.805 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd574138c80 con 0x7fd5741430c0 2024-03-25T19:45:17.805 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5740b1920 con 0x7fd57413f710 2024-03-25T19:45:17.805 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5740b05e0 con 0x7fd5740ae7a0 2024-03-25T19:45:17.805 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:17.805 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.803+0000 7fd57aed9640 10 monclient(hunting): authenticate will time out at 4488.944753s 2024-03-25T19:45:17.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.813+0000 7fd573fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd5741430c0 0x7fd574143550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37916/0 (socket says 172.21.15.110:37916) 2024-03-25T19:45:17.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.813+0000 7fd573fff640 1 -- 172.21.15.110:0/580776294 learned_addr learned my addr 172.21.15.110:0/580776294 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:17.815 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.815+0000 7fd5717fa640 10 client.?.objecter ms_handle_connect 0x7fd57413f710 2024-03-25T19:45:17.815 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.815+0000 7fd5717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.816 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.817+0000 7fd5717fa640 10 client.?.objecter ms_handle_connect 0x7fd5741430c0 2024-03-25T19:45:17.816 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.817+0000 7fd5717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.816 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.817+0000 7fd5717fa640 10 client.?.objecter ms_handle_connect 0x7fd5740ae7a0 2024-03-25T19:45:17.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.817+0000 7fd5717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.840+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207982689 0 0) 0x7fd574138c80 con 0x7fd5741430c0 2024-03-25T19:45:17.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.840+0000 7fd5717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.840+0000 7fd5717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.840+0000 7fd5717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.840+0000 7fd5717fa640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:45:17.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.840+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd558004eb0 con 0x7fd5741430c0 2024-03-25T19:45:17.846 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:25.130 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.897 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:25.181 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.904+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130209448 0 0) 0x7fd558004eb0 con 0x7fd5741430c0 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.904+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd574138c80 con 0x7fd5741430c0 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.904+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd560001a30 con 0x7fd5741430c0 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.904+0000 7fd5717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.904+0000 7fd5717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.904+0000 7fd5717fa640 10 monclient(hunting): dump: 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:17.904 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:17.905 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:17.905 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:17.905 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:17.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.904+0000 7fd5717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.917+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2379013686 0 0) 0x7fd574138c80 con 0x7fd5741430c0 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.917+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 >> v1:172.21.15.110:6790/0 conn(0x7fd5740ae7a0 legacy=0x7fd5740afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.917+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 >> v1:172.21.15.178:6789/0 conn(0x7fd57413f710 legacy=0x7fd5740adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.917+0000 7fd5717fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.917+0000 7fd5717fa640 1 monclient: found mon.a 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.917+0000 7fd5717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.917+0000 7fd5717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.917+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740af2c0 con 0x7fd5741430c0 2024-03-25T19:45:17.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd5717fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd5717fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd5717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 10 monclient: _renew_subs 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5741e34b0 con 0x7fd5741430c0 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 10 monclient: _renew_subs 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5741e3820 con 0x7fd5741430c0 2024-03-25T19:45:17.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.918+0000 7fd57aed9640 1 librados: init done 2024-03-25T19:45:17.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.919+0000 7fd57aed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.919+0000 7fd57aed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.919+0000 7fd57aed9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:17.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:17.919+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5740b3ef0 con 0x7fd5741430c0 2024-03-25T19:45:17.947 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:25.201 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.952 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:25.207 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.954 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:25.238 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.961 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:25.245 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.991 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:25.245 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:18.005 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:25.289 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:18.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.142+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd560001cb0 con 0x7fd5741430c0 2024-03-25T19:45:18.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.142+0000 7fd5717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:18.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.142+0000 7fd572ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:18.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.205+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd560003a60 con 0x7fd5741430c0 2024-03-25T19:45:18.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.206+0000 7fd5717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:18.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.206+0000 7fd5717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:18.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.206+0000 7fd5717fa640 10 monclient: dump: 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:18.206 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:18.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.206+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fd560003c40 con 0x7fd5741430c0 2024-03-25T19:45:18.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.207+0000 7fd578c75640 10 monclient: get_auth_request con 0x7fd55806dd20 auth_method 0 2024-03-25T19:45:18.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.207+0000 7fd5717fa640 10 client.4437.objecter ms_handle_connect 0x7fd55806dd20 2024-03-25T19:45:18.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.213+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fd5600548a0 con 0x7fd5741430c0 2024-03-25T19:45:18.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.213+0000 7fd5717fa640 10 client.4437.objecter ms_dispatch 0x7fd574000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:18.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.213+0000 7fd5717fa640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:18.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.213+0000 7fd5717fa640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:18.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.213+0000 7fd5717fa640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T19:45:18.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.213+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5600501e0 con 0x7fd5741430c0 2024-03-25T19:45:18.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.213+0000 7fd5717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:18.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.213+0000 7fd5717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:18.214 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:25.498 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:18.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.455+0000 7fd57aed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.455+0000 7fd57aed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.455+0000 7fd57aed9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:18.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.455+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5740627b0 con 0x7fd5741430c0 2024-03-25T19:45:18.475 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.475+0000 7fd5717fa640 1 -- 172.21.15.110:0/580776294 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fd5600522e0 con 0x7fd5741430c0 2024-03-25T19:45:18.475 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.475+0000 7fd5717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.475 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.475+0000 7fd5717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:18.476 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:45:18.476 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:18.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 10 librados: watch_flush enter 2024-03-25T19:45:18.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 10 librados: watch_flush exit 2024-03-25T19:45:18.478 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 10 client.4437.objecter successfully canceled tick 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 >> v1:172.21.15.178:6800/1132739058 conn(0x7fd55806dd20 legacy=0x7fd5580701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 10 monclient: shutdown 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 >> v1:172.21.15.110:6789/0 conn(0x7fd5741430c0 legacy=0x7fd574143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 shutdown_connections 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.478+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 >> 172.21.15.110:0/580776294 conn(0x7fd5740b7ea0 msgr2=0x7fd574074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.479+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 shutdown_connections 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.479+0000 7fd57aed9640 1 -- 172.21.15.110:0/580776294 wait complete. 2024-03-25T19:45:18.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:18.479+0000 7fd57aed9640 1 librados: shutdown 2024-03-25T19:45:18.491 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:25.297 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:25.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.416+0000 7fac0100e640 10 monclient: get_monmap_and_config 2024-03-25T19:45:25.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.416+0000 7fac0100e640 10 monclient: build_initial_monmap 2024-03-25T19:45:25.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.416+0000 7fac0100e640 10 monclient: monmap: 2024-03-25T19:45:25.416 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:25.416 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:25.416 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:45:25.417750+0000 2024-03-25T19:45:25.416 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:45:25.417750+0000 2024-03-25T19:45:25.417 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:25.417 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:25.417 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:25.417 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:45:25.417 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:45:25.417 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:25.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.417+0000 7fac0100e640 1 Processor -- start 2024-03-25T19:45:25.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.417+0000 7fac0100e640 1 -- start start 2024-03-25T19:45:25.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.417+0000 7fac0100e640 10 monclient: init 2024-03-25T19:45:25.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:25.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:25.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 10 monclient(hunting): picked mon.noname-b con 0x7fabfc13fc60 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:25.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 10 monclient(hunting): picked mon.noname-a con 0x7fabfc070570 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:25.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 10 monclient(hunting): picked mon.noname-c con 0x7fabfc0727a0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:25.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc13c9f0 con 0x7fabfc070570 2024-03-25T19:45:25.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc06fac0 con 0x7fabfc0727a0 2024-03-25T19:45:25.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc0715d0 con 0x7fabfc13fc60 2024-03-25T19:45:25.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:25.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.418+0000 7fac0100e640 10 monclient(hunting): authenticate will time out at 4496.559394s 2024-03-25T19:45:25.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.425+0000 7fabfa59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fabfc070570 0x7fabfc072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53966/0 (socket says 172.21.15.110:53966) 2024-03-25T19:45:25.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.425+0000 7fabfa59c640 1 -- 172.21.15.110:0/2808447966 learned_addr learned my addr 172.21.15.110:0/2808447966 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:25.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.440+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444197956 0 0) 0x7fabfc13c9f0 con 0x7fabfc070570 2024-03-25T19:45:25.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.440+0000 7fabf9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.440+0000 7fabf9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.440+0000 7fabf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.440+0000 7fabf9d9b640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:45:25.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.440+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe4003520 con 0x7fabfc070570 2024-03-25T19:45:25.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.446+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960564363 0 0) 0x7fabfc06fac0 con 0x7fabfc0727a0 2024-03-25T19:45:25.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.446+0000 7fabf9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.446+0000 7fabf9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.446+0000 7fabf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.446+0000 7fabf9d9b640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:45:25.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.446+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabfc13c9f0 con 0x7fabfc0727a0 2024-03-25T19:45:25.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.459+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945940092 0 0) 0x7fabfc0715d0 con 0x7fabfc13fc60 2024-03-25T19:45:25.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.459+0000 7fabf9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.459+0000 7fabf9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.459+0000 7fabf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.459+0000 7fabf9d9b640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:45:25.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.459+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabfc06fac0 con 0x7fabfc13fc60 2024-03-25T19:45:25.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463071539 0 0) 0x7fabe4003520 con 0x7fabfc070570 2024-03-25T19:45:25.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 >> v1:172.21.15.110:6790/0 conn(0x7fabfc13fc60 legacy=0x7fabfc06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 >> v1:172.21.15.178:6789/0 conn(0x7fabfc0727a0 legacy=0x7fabfc14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 1 monclient: found mon.noname-a 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc0718b0 con 0x7fabfc070570 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fabf0001690 con 0x7fabfc070570 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.460+0000 7fabf9d9b640 10 monclient: dump: 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.461 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.462 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.462 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:25.462 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:25.462 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:25.462 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:25.462 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:25.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.461+0000 7fac0100e640 5 monclient: authenticate success, global_id 4440 2024-03-25T19:45:25.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.461+0000 7fac0100e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.471+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabf00017f0 con 0x7fabfc070570 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.471+0000 7fabf9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.471+0000 7fabf9d9b640 1 -- 172.21.15.110:0/2808447966 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fabf0001c00 con 0x7fabfc070570 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.471+0000 7fac0100e640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.471+0000 7fac0100e640 4 set_mon_vals no callback set 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.471+0000 7fabf9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.471+0000 7fabf9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.471+0000 7fabf9d9b640 10 monclient: dump: 2024-03-25T19:45:25.471 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:25.472 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:25.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.472+0000 7fac0100e640 10 monclient: shutdown 2024-03-25T19:45:25.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.472+0000 7fac0100e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:25.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.472+0000 7fac0100e640 1 -- 172.21.15.110:0/2808447966 >> v1:172.21.15.110:6789/0 conn(0x7fabfc070570 legacy=0x7fabfc072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.472+0000 7fac0100e640 1 -- 172.21.15.110:0/2808447966 shutdown_connections 2024-03-25T19:45:25.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.472+0000 7fac0100e640 1 -- 172.21.15.110:0/2808447966 >> 172.21.15.110:0/2808447966 conn(0x7fabfc0b4a80 msgr2=0x7fabfc0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.472+0000 7fac0100e640 1 -- 172.21.15.110:0/2808447966 shutdown_connections 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.473+0000 7fac0100e640 1 -- 172.21.15.110:0/2808447966 wait complete. 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 10 monclient: build_initial_monmap 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 10 monclient: monmap: 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:45:25.474 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:25.475 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 1 librados: starting msgr at 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 1 librados: starting objecter 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 1 Processor -- start 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 1 -- start start 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 1 librados: setting wanted keys 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 1 librados: calling monclient init 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.474+0000 7fac0100e640 10 monclient: init 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 10 monclient(hunting): picked mon.noname-a con 0x7fabfc070570 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 10 monclient(hunting): picked mon.noname-b con 0x7fabfc0727a0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 10 monclient(hunting): picked mon.noname-c con 0x7fabfc13fc60 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc135400 con 0x7fabfc070570 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc136030 con 0x7fabfc0727a0 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc143390 con 0x7fabfc13fc60 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:25.476 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.475+0000 7fac0100e640 10 monclient(hunting): authenticate will time out at 4496.616466s 2024-03-25T19:45:25.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.481+0000 7fabfad9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fabfc070570 0x7fabfc134ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54008/0 (socket says 172.21.15.110:54008) 2024-03-25T19:45:25.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.481+0000 7fabfad9d640 1 -- 172.21.15.110:0/3489091738 learned_addr learned my addr 172.21.15.110:0/3489091738 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:25.481 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.482+0000 7fabe37fe640 10 client.?.objecter ms_handle_connect 0x7fabfc070570 2024-03-25T19:45:25.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.482+0000 7fabe37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.494+0000 7fabe37fe640 10 client.?.objecter ms_handle_connect 0x7fabfc0727a0 2024-03-25T19:45:25.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.494+0000 7fabe37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.495+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082313280 0 0) 0x7fabfc135400 con 0x7fabfc070570 2024-03-25T19:45:25.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.495+0000 7fabe37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.495+0000 7fabe37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.495+0000 7fabe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.495+0000 7fabe37fe640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:45:25.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.495+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe8004eb0 con 0x7fabfc070570 2024-03-25T19:45:25.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.498+0000 7fabe37fe640 10 client.?.objecter ms_handle_connect 0x7fabfc13fc60 2024-03-25T19:45:25.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.498+0000 7fabe37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.507+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340829600 0 0) 0x7fabfc136030 con 0x7fabfc0727a0 2024-03-25T19:45:25.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.507+0000 7fabe37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.507+0000 7fabe37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.507+0000 7fabe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.507+0000 7fabe37fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:45:25.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.507+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabfc135400 con 0x7fabfc0727a0 2024-03-25T19:45:25.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.509+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1039908144 0 0) 0x7fabe8004eb0 con 0x7fabfc070570 2024-03-25T19:45:25.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.509+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabfc136030 con 0x7fabfc070570 2024-03-25T19:45:25.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.509+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fabec001970 con 0x7fabfc070570 2024-03-25T19:45:25.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.509+0000 7fabe37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.509+0000 7fabe37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:25.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.509+0000 7fabe37fe640 10 monclient(hunting): dump: 2024-03-25T19:45:25.509 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:25.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.509+0000 7fabe37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:25.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.517+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 664795581 0 0) 0x7fabfc135400 con 0x7fabfc0727a0 2024-03-25T19:45:25.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.518+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe8004eb0 con 0x7fabfc0727a0 2024-03-25T19:45:25.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.518+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fabf0001b00 con 0x7fabfc0727a0 2024-03-25T19:45:25.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.518+0000 7fabe37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.518+0000 7fabe37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:25.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.518+0000 7fabe37fe640 10 monclient(hunting): dump: 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:25.518 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:25.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 513844414 0 0) 0x7fabfc136030 con 0x7fabfc070570 2024-03-25T19:45:25.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 >> v1:172.21.15.110:6790/0 conn(0x7fabfc13fc60 legacy=0x7fabfc1e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 >> v1:172.21.15.178:6789/0 conn(0x7fabfc0727a0 legacy=0x7fabfc135920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:25.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 1 monclient: found mon.a 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc1dab40 con 0x7fabfc070570 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fabe37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 5 monclient: authenticate success, global_id 4443 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 10 monclient: _renew_subs 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc1dadf0 con 0x7fabfc070570 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 10 monclient: _renew_subs 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc13a480 con 0x7fabfc070570 2024-03-25T19:45:25.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.520+0000 7fac0100e640 1 librados: init done 2024-03-25T19:45:25.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.522+0000 7fac0100e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.522+0000 7fac0100e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.522+0000 7fac0100e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:25.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.522+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc0627b0 con 0x7fabfc070570 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.853+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabec003610 con 0x7fabfc070570 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.853+0000 7fabe37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.853+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fabec003a20 con 0x7fabfc070570 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.853+0000 7fabe37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.853+0000 7fabe37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.853+0000 7fabe37fe640 10 monclient: dump: 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.853+0000 7fabf959a640 4 set_mon_vals no callback set 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.853+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fabec003c40 con 0x7fabfc070570 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.854+0000 7fabfa59c640 10 monclient: get_auth_request con 0x7fabe806de90 auth_method 0 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.854+0000 7fabe37fe640 10 client.4443.objecter ms_handle_connect 0x7fabe806de90 2024-03-25T19:45:25.860 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.860+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fabec0547d0 con 0x7fabfc070570 2024-03-25T19:45:25.860 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.860+0000 7fabe37fe640 10 client.4443.objecter ms_dispatch 0x7fabfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:25.860 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.860+0000 7fabe37fe640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:25.860 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.860+0000 7fabe37fe640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:25.860 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.860+0000 7fabe37fe640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T19:45:25.860 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.860+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabec0502e0 con 0x7fabfc070570 2024-03-25T19:45:25.860 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.860+0000 7fabe37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.860 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:25.860+0000 7fabe37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:26.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.067+0000 7fac0100e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:26.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.067+0000 7fac0100e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:26.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.067+0000 7fac0100e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:26.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.067+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabfc1361c0 con 0x7fabfc070570 2024-03-25T19:45:26.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.140+0000 7fabe37fe640 1 -- 172.21.15.110:0/3489091738 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fabec052210 con 0x7fabfc070570 2024-03-25T19:45:26.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.140+0000 7fabe37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:26.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.140+0000 7fabe37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:26.140 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:45:26.140 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:26.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 10 librados: watch_flush enter 2024-03-25T19:45:26.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 10 librados: watch_flush exit 2024-03-25T19:45:26.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T19:45:26.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 10 client.4443.objecter successfully canceled tick 2024-03-25T19:45:26.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 >> v1:172.21.15.178:6800/1132739058 conn(0x7fabe806de90 legacy=0x7fabe8070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 10 monclient: shutdown 2024-03-25T19:45:26.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:26.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 >> v1:172.21.15.110:6789/0 conn(0x7fabfc070570 legacy=0x7fabfc134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 shutdown_connections 2024-03-25T19:45:26.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 >> 172.21.15.110:0/3489091738 conn(0x7fabfc0b4a80 msgr2=0x7fabfc0b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:26.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 shutdown_connections 2024-03-25T19:45:26.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 1 -- 172.21.15.110:0/3489091738 wait complete. 2024-03-25T19:45:26.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:26.142+0000 7fac0100e640 1 librados: shutdown 2024-03-25T19:45:26.155 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:32.963 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:33.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.078+0000 7fcaac025640 10 monclient: get_monmap_and_config 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.078+0000 7fcaac025640 10 monclient: build_initial_monmap 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.078+0000 7fcaac025640 10 monclient: monmap: 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:45:33.079011+0000 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:45:33.079011+0000 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.078+0000 7fcaac025640 1 Processor -- start 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.079+0000 7fcaac025640 1 -- start start 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.079+0000 7fcaac025640 10 monclient: init 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.079+0000 7fcaac025640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.079+0000 7fcaac025640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.079+0000 7fcaac025640 10 monclient(hunting): picked mon.noname-c con 0x7fcaa40a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.080+0000 7fcaac025640 10 monclient(hunting): picked mon.noname-a con 0x7fcaa413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.080+0000 7fcaac025640 10 monclient(hunting): picked mon.noname-b con 0x7fcaa41430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.080+0000 7fcaac025640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa4142520 con 0x7fcaa413f710 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.080+0000 7fcaac025640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa40b28a0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.080+0000 7fcaac025640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa40b2a30 con 0x7fcaa41430c0 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.080+0000 7fcaac025640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:33.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.080+0000 7fcaac025640 10 monclient(hunting): authenticate will time out at 4504.221037s 2024-03-25T19:45:33.088 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.088+0000 7fcaa95c0640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcaa413f710 0x7fcaa41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58146/0 (socket says 172.21.15.110:58146) 2024-03-25T19:45:33.088 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.089+0000 7fcaa95c0640 1 -- 172.21.15.110:0/2215302313 learned_addr learned my addr 172.21.15.110:0/2215302313 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:33.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.097+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338575297 0 0) 0x7fcaa4142520 con 0x7fcaa413f710 2024-03-25T19:45:33.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.097+0000 7fcaa8dbf640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.097+0000 7fcaa8dbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.097+0000 7fcaa8dbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.097+0000 7fcaa8dbf640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:45:33.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.097+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca90003520 con 0x7fcaa413f710 2024-03-25T19:45:33.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.111+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321280418 0 0) 0x7fcaa40b28a0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.111+0000 7fcaa8dbf640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.111+0000 7fcaa8dbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.111+0000 7fcaa8dbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.111+0000 7fcaa8dbf640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:45:33.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.111+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa4142520 con 0x7fcaa40a98f0 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1522612823 0 0) 0x7fca90003520 con 0x7fcaa413f710 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 >> v1:172.21.15.110:6790/0 conn(0x7fcaa41430c0 legacy=0x7fcaa414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 >> v1:172.21.15.178:6789/0 conn(0x7fcaa40a98f0 legacy=0x7fcaa40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 10 monclient: _finish_hunting 0 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 1 monclient: found mon.noname-a 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa40b44c0 con 0x7fcaa413f710 2024-03-25T19:45:33.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 10 monclient: _finish_auth 0 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 10 monclient: _check_auth_tickets 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fca94001690 con 0x7fcaa413f710 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaa8dbf640 10 monclient: dump: 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:33.114 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:33.115 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:33.115 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:33.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaac025640 5 monclient: authenticate success, global_id 4446 2024-03-25T19:45:33.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.113+0000 7fcaac025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.122+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca940017f0 con 0x7fcaa413f710 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.122+0000 7fcaa8dbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.122+0000 7fcaa8dbf640 1 -- 172.21.15.110:0/2215302313 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fca94001c00 con 0x7fcaa413f710 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.122+0000 7fcaac025640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.122+0000 7fcaac025640 4 set_mon_vals no callback set 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.122+0000 7fcaa8dbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.122+0000 7fcaa8dbf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.122+0000 7fcaa8dbf640 10 monclient: dump: 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:33.122 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:33.123 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.123 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.123 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.123 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:33.123 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:33.123 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:33.123 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:33.123 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:33.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.123+0000 7fcaac025640 10 monclient: shutdown 2024-03-25T19:45:33.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.123+0000 7fcaac025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:33.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.123+0000 7fcaac025640 1 -- 172.21.15.110:0/2215302313 >> v1:172.21.15.110:6789/0 conn(0x7fcaa413f710 legacy=0x7fcaa41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.123+0000 7fcaac025640 1 -- 172.21.15.110:0/2215302313 shutdown_connections 2024-03-25T19:45:33.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.123+0000 7fcaac025640 1 -- 172.21.15.110:0/2215302313 >> 172.21.15.110:0/2215302313 conn(0x7fcaa40b7ea0 msgr2=0x7fcaa40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:33.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.123+0000 7fcaac025640 1 -- 172.21.15.110:0/2215302313 shutdown_connections 2024-03-25T19:45:33.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.123+0000 7fcaac025640 1 -- 172.21.15.110:0/2215302313 wait complete. 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.124+0000 7fcaac025640 10 monclient: build_initial_monmap 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.124+0000 7fcaac025640 10 monclient: monmap: 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.124+0000 7fcaac025640 1 librados: starting msgr at 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.124+0000 7fcaac025640 1 librados: starting objecter 2024-03-25T19:45:33.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.125+0000 7fcaac025640 1 Processor -- start 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.125+0000 7fcaac025640 1 -- start start 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.125+0000 7fcaac025640 1 librados: setting wanted keys 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.125+0000 7fcaac025640 1 librados: calling monclient init 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.125+0000 7fcaac025640 10 monclient: init 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.125+0000 7fcaac025640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.125+0000 7fcaac025640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.126+0000 7fcaac025640 10 monclient(hunting): picked mon.noname-b con 0x7fcaa40a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.126+0000 7fcaac025640 10 monclient(hunting): picked mon.noname-a con 0x7fcaa413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.126+0000 7fcaac025640 10 monclient(hunting): picked mon.noname-c con 0x7fcaa41430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.126+0000 7fcaac025640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa4070550 con 0x7fcaa413f710 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.126+0000 7fcaac025640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa4139a30 con 0x7fcaa40a98f0 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.126+0000 7fcaac025640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa4070780 con 0x7fcaa41430c0 2024-03-25T19:45:33.127 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.126+0000 7fcaac025640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:33.128 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.126+0000 7fcaac025640 10 monclient(hunting): authenticate will time out at 4504.267089s 2024-03-25T19:45:33.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.137+0000 7fcaa9dc1640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fcaa40a98f0 0x7fcaa41df420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:39846/0 (socket says 172.21.15.110:39846) 2024-03-25T19:45:33.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.137+0000 7fcaa9dc1640 1 -- 172.21.15.110:0/3044517911 learned_addr learned my addr 172.21.15.110:0/3044517911 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:33.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.138+0000 7fca9a7fc640 10 client.?.objecter ms_handle_connect 0x7fcaa40a98f0 2024-03-25T19:45:33.138 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.138+0000 7fca9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.141+0000 7fca9a7fc640 10 client.?.objecter ms_handle_connect 0x7fcaa41430c0 2024-03-25T19:45:33.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.141+0000 7fca9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.148+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155119380 0 0) 0x7fcaa4139a30 con 0x7fcaa40a98f0 2024-03-25T19:45:33.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.148+0000 7fca9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.148+0000 7fca9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.148+0000 7fca9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.148+0000 7fca9a7fc640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:45:33.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.148+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca88004eb0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.149+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181491975 0 0) 0x7fcaa4070780 con 0x7fcaa41430c0 2024-03-25T19:45:33.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.149+0000 7fca9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.149+0000 7fca9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.149+0000 7fca9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.149+0000 7fca9a7fc640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:45:33.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.149+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa4139a30 con 0x7fcaa41430c0 2024-03-25T19:45:33.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.153+0000 7fca9a7fc640 10 client.?.objecter ms_handle_connect 0x7fcaa413f710 2024-03-25T19:45:33.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.153+0000 7fca9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.168+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952461493 0 0) 0x7fcaa4139a30 con 0x7fcaa41430c0 2024-03-25T19:45:33.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.168+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa4070780 con 0x7fcaa41430c0 2024-03-25T19:45:33.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.168+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fca9c001810 con 0x7fcaa41430c0 2024-03-25T19:45:33.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.168+0000 7fca9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.168+0000 7fca9a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:33.168 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.168+0000 7fca9a7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:33.168 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:33.169 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.168+0000 7fca9a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:33.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.171+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403095020 0 0) 0x7fcaa4070550 con 0x7fcaa413f710 2024-03-25T19:45:33.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.171+0000 7fca9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.171+0000 7fca9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.171+0000 7fca9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.171+0000 7fca9a7fc640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:45:33.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.172+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa4139a30 con 0x7fcaa413f710 2024-03-25T19:45:33.176 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.176+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1974137983 0 0) 0x7fca88004eb0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.176 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.176+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa4070550 con 0x7fcaa40a98f0 2024-03-25T19:45:33.176 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.176+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcaa0001d90 con 0x7fcaa40a98f0 2024-03-25T19:45:33.176 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.176+0000 7fca9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.176+0000 7fca9a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.176+0000 7fca9a7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:33.177 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3836997164 0 0) 0x7fcaa4070550 con 0x7fcaa40a98f0 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 >> v1:172.21.15.110:6790/0 conn(0x7fcaa41430c0 legacy=0x7fcaa4139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 >> v1:172.21.15.110:6789/0 conn(0x7fcaa413f710 legacy=0x7fcaa4138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 1 monclient: found mon.b 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa406e650 con 0x7fcaa40a98f0 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:33.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fca9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 10 monclient: _renew_subs 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa4070ba0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 10 monclient: _renew_subs 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa41dfd10 con 0x7fcaa40a98f0 2024-03-25T19:45:33.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.187+0000 7fcaac025640 1 librados: init done 2024-03-25T19:45:33.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.188+0000 7fcaac025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.188+0000 7fcaac025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.188+0000 7fcaac025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:45:33.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.188+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa41df0b0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.459+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaa0001fd0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.459+0000 7fca9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.459+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fcaa0003c40 con 0x7fcaa40a98f0 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.459+0000 7fca9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.459+0000 7fca9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.459+0000 7fca9a7fc640 10 monclient: dump: 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:33.459 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:33.460 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:33.460 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:33.460 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:33.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.459+0000 7fca9bfff640 4 set_mon_vals no callback set 2024-03-25T19:45:33.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.460+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fcaa0003e20 con 0x7fcaa40a98f0 2024-03-25T19:45:33.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.460+0000 7fcaa95c0640 10 monclient: get_auth_request con 0x7fca8806df50 auth_method 0 2024-03-25T19:45:33.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.460+0000 7fca9a7fc640 10 client.4459.objecter ms_handle_connect 0x7fca8806df50 2024-03-25T19:45:33.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.464+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fcaa0054af0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.464+0000 7fca9a7fc640 10 client.4459.objecter ms_dispatch 0x7fcaa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:33.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.464+0000 7fca9a7fc640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:33.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.464+0000 7fca9a7fc640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:33.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.464+0000 7fca9a7fc640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T19:45:33.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.465+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcaa0050600 con 0x7fcaa40a98f0 2024-03-25T19:45:33.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.465+0000 7fca9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.465+0000 7fca9a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:33.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.716+0000 7fcaac025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.716+0000 7fcaac025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.716+0000 7fcaac025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:45:33.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.716+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcaa40627b0 con 0x7fcaa40a98f0 2024-03-25T19:45:33.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.832+0000 7fca9a7fc640 1 -- 172.21.15.110:0/3044517911 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fcaa0052530 con 0x7fcaa40a98f0 2024-03-25T19:45:33.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.832+0000 7fca9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.832 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.832+0000 7fca9a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:33.832 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:45:33.832 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:33.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 10 librados: watch_flush enter 2024-03-25T19:45:33.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 10 librados: watch_flush exit 2024-03-25T19:45:33.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T19:45:33.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 10 client.4459.objecter successfully canceled tick 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 >> v1:172.21.15.178:6800/1132739058 conn(0x7fca8806df50 legacy=0x7fca88070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 10 monclient: shutdown 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 >> v1:172.21.15.178:6789/0 conn(0x7fcaa40a98f0 legacy=0x7fcaa41df420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 shutdown_connections 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 >> 172.21.15.110:0/3044517911 conn(0x7fcaa40b7ea0 msgr2=0x7fcaa4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 shutdown_connections 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 1 -- 172.21.15.110:0/3044517911 wait complete. 2024-03-25T19:45:33.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:33.834+0000 7fcaac025640 1 librados: shutdown 2024-03-25T19:45:33.844 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:40.651 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:40.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.764+0000 7f645b9e6640 10 monclient: get_monmap_and_config 2024-03-25T19:45:40.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.764+0000 7f645b9e6640 10 monclient: build_initial_monmap 2024-03-25T19:45:40.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.764+0000 7f645b9e6640 10 monclient: monmap: 2024-03-25T19:45:40.764 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:40.764 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:40.764 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:45:40.765615+0000 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:45:40.765615+0000 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.765+0000 7f645b9e6640 1 Processor -- start 2024-03-25T19:45:40.765 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.765+0000 7f645b9e6640 1 -- start start 2024-03-25T19:45:40.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.765+0000 7f645b9e6640 10 monclient: init 2024-03-25T19:45:40.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 10 monclient(hunting): picked mon.noname-a con 0x7f64540a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 10 monclient(hunting): picked mon.noname-c con 0x7f645413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 10 monclient(hunting): picked mon.noname-b con 0x7f64541430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6454142520 con 0x7f64540a98f0 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64540b28a0 con 0x7f645413f710 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64540b2a30 con 0x7f64541430c0 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:40.767 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.766+0000 7f645b9e6640 10 monclient(hunting): authenticate will time out at 4511.907623s 2024-03-25T19:45:40.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.779+0000 7f6459f83640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f64541430c0 0x7f645414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:47106/0 (socket says 172.21.15.110:47106) 2024-03-25T19:45:40.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.779+0000 7f6459f83640 1 -- 172.21.15.110:0/3674702690 learned_addr learned my addr 172.21.15.110:0/3674702690 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:40.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.794+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956902768 0 0) 0x7f64540b2a30 con 0x7f64541430c0 2024-03-25T19:45:40.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.794+0000 7f644bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.794+0000 7f644bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.794+0000 7f644bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.794+0000 7f644bfff640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:45:40.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.794+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6440003520 con 0x7f64541430c0 2024-03-25T19:45:40.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.796+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247755897 0 0) 0x7f64540b28a0 con 0x7f645413f710 2024-03-25T19:45:40.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.796+0000 7f644bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.796+0000 7f644bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.796+0000 7f644bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.796+0000 7f644bfff640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:45:40.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.796+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64540b2a30 con 0x7f645413f710 2024-03-25T19:45:40.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.806+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197770586 0 0) 0x7f6454142520 con 0x7f64540a98f0 2024-03-25T19:45:40.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.806+0000 7f644bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.806+0000 7f644bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.806+0000 7f644bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.806+0000 7f644bfff640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:45:40.806 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.806+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64540b28a0 con 0x7f64540a98f0 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 572681224 0 0) 0x7f64540b2a30 con 0x7f645413f710 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 >> v1:172.21.15.110:6790/0 conn(0x7f64541430c0 legacy=0x7f645414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 >> v1:172.21.15.110:6789/0 conn(0x7f64540a98f0 legacy=0x7f64540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 1 monclient: found mon.noname-c 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64540b44c0 con 0x7f645413f710 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 10 monclient: _finish_auth 0 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6444001670 con 0x7f645413f710 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f644bfff640 10 monclient: dump: 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f645b9e6640 5 monclient: authenticate success, global_id 4462 2024-03-25T19:45:40.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.811+0000 7f645b9e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.831+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64440017d0 con 0x7f645413f710 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.831+0000 7f644bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.831+0000 7f644bfff640 1 -- 172.21.15.110:0/3674702690 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6444001be0 con 0x7f645413f710 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.831+0000 7f645b9e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.831+0000 7f645b9e6640 4 set_mon_vals no callback set 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.831+0000 7f644bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.831+0000 7f644bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.831+0000 7f644bfff640 10 monclient: dump: 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.831 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:40.832 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:40.832 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:40.832 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:40.832 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:40.833 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.832+0000 7f645b9e6640 10 monclient: shutdown 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.832+0000 7f645b9e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.832+0000 7f645b9e6640 1 -- 172.21.15.110:0/3674702690 >> v1:172.21.15.178:6789/0 conn(0x7f645413f710 legacy=0x7f64541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.832+0000 7f645b9e6640 1 -- 172.21.15.110:0/3674702690 shutdown_connections 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.832+0000 7f645b9e6640 1 -- 172.21.15.110:0/3674702690 >> 172.21.15.110:0/3674702690 conn(0x7f64540b7ea0 msgr2=0x7f64540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.832+0000 7f645b9e6640 1 -- 172.21.15.110:0/3674702690 shutdown_connections 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.832+0000 7f645b9e6640 1 -- 172.21.15.110:0/3674702690 wait complete. 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.833+0000 7f645b9e6640 10 monclient: build_initial_monmap 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.833+0000 7f645b9e6640 10 monclient: monmap: 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 1 librados: starting msgr at 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 1 librados: starting objecter 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 1 Processor -- start 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 1 -- start start 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 1 librados: setting wanted keys 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 1 librados: calling monclient init 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 10 monclient: init 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 10 monclient(hunting): picked mon.noname-b con 0x7f64540a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 10 monclient(hunting): picked mon.noname-c con 0x7f645413f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 10 monclient(hunting): picked mon.noname-a con 0x7f64541430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:40.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64541d7830 con 0x7f64541430c0 2024-03-25T19:45:40.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.834+0000 7f645b9e6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64541d51f0 con 0x7f64540a98f0 2024-03-25T19:45:40.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.835+0000 7f645b9e6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64541d83d0 con 0x7f645413f710 2024-03-25T19:45:40.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.835+0000 7f645b9e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:40.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.835+0000 7f645b9e6640 10 monclient(hunting): authenticate will time out at 4511.975866s 2024-03-25T19:45:40.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.844+0000 7f6458f81640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f645413f710 0x7f64541387e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:47176/0 (socket says 172.21.15.110:47176) 2024-03-25T19:45:40.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.844+0000 7f6458f81640 1 -- 172.21.15.110:0/2106985209 learned_addr learned my addr 172.21.15.110:0/2106985209 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.850+0000 7f6449ffb640 10 client.?.objecter ms_handle_connect 0x7f645413f710 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.850+0000 7f6449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.850+0000 7f6449ffb640 10 client.?.objecter ms_handle_connect 0x7f64541430c0 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.850+0000 7f6449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.852+0000 7f6449ffb640 10 client.?.objecter ms_handle_connect 0x7f64540a98f0 2024-03-25T19:45:40.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.852+0000 7f6449ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.859+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089616105 0 0) 0x7f64541d7830 con 0x7f64541430c0 2024-03-25T19:45:40.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.859+0000 7f6449ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.859+0000 7f6449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.859+0000 7f6449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.859+0000 7f6449ffb640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:45:40.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.859+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6438004eb0 con 0x7f64541430c0 2024-03-25T19:45:40.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.867+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896545052 0 0) 0x7f64541d51f0 con 0x7f64540a98f0 2024-03-25T19:45:40.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.867+0000 7f6449ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.867+0000 7f6449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.867+0000 7f6449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.867+0000 7f6449ffb640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:45:40.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.867+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64541d7830 con 0x7f64540a98f0 2024-03-25T19:45:40.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.868+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151046319 0 0) 0x7f64541d83d0 con 0x7f645413f710 2024-03-25T19:45:40.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.868+0000 7f6449ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.868+0000 7f6449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.868+0000 7f6449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.868+0000 7f6449ffb640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:45:40.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.868+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64541d51f0 con 0x7f645413f710 2024-03-25T19:45:40.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.878+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2579633699 0 0) 0x7f64541d7830 con 0x7f64540a98f0 2024-03-25T19:45:40.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.879+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64541d83d0 con 0x7f64540a98f0 2024-03-25T19:45:40.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.879+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6450001870 con 0x7f64540a98f0 2024-03-25T19:45:40.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.879+0000 7f6449ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.879+0000 7f6449ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.879+0000 7f6449ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:40.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.879+0000 7f6449ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.880+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2278677950 0 0) 0x7f6438004eb0 con 0x7f64541430c0 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.880+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64541d7830 con 0x7f64541430c0 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.880+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f644c001af0 con 0x7f64541430c0 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.880+0000 7f6449ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.880+0000 7f6449ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.880+0000 7f6449ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:40.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3060712209 0 0) 0x7f64541d51f0 con 0x7f645413f710 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6438004eb0 con 0x7f645413f710 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6444001630 con 0x7f645413f710 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:40.893 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2385257055 0 0) 0x7f64541d7830 con 0x7f64541430c0 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 >> v1:172.21.15.110:6790/0 conn(0x7f645413f710 legacy=0x7f64541387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.891+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 >> v1:172.21.15.178:6789/0 conn(0x7f64540a98f0 legacy=0x7f64541d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f6449ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f6449ffb640 1 monclient: found mon.a 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f6449ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f6449ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454138ef0 con 0x7f64541430c0 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f6449ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f6449ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f6449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:45:40.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 10 monclient: _renew_subs 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6454137ad0 con 0x7f64541430c0 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 10 monclient: _renew_subs 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64541e3550 con 0x7f64541430c0 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.892+0000 7f645b9e6640 1 librados: init done 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.893+0000 7f645b9e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.893+0000 7f645b9e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.893+0000 7f645b9e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:40.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:40.893+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64540abe50 con 0x7f64541430c0 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.194+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f644c001fe0 con 0x7f64541430c0 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.194+0000 7f6449ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.194+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f644c004d00 con 0x7f64541430c0 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.194+0000 7f6449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.194+0000 7f6449ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.194+0000 7f6449ffb640 10 monclient: dump: 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:41.194 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:41.195 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:41.195 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:41.195 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:41.195 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:41.195 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:41.195 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.194+0000 7f644b7fe640 4 set_mon_vals no callback set 2024-03-25T19:45:41.195 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.195+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f644c01acd0 con 0x7f64541430c0 2024-03-25T19:45:41.196 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.195+0000 7f6459782640 10 monclient: get_auth_request con 0x7f643806e0c0 auth_method 0 2024-03-25T19:45:41.196 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.195+0000 7f6449ffb640 10 client.4455.objecter ms_handle_connect 0x7f643806e0c0 2024-03-25T19:45:41.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.201+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f644c0549e0 con 0x7f64541430c0 2024-03-25T19:45:41.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.201+0000 7f6449ffb640 10 client.4455.objecter ms_dispatch 0x7f6454000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.201+0000 7f6449ffb640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.201+0000 7f6449ffb640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.201+0000 7f6449ffb640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T19:45:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.201+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f644c050320 con 0x7f64541430c0 2024-03-25T19:45:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.201+0000 7f6449ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.201+0000 7f6449ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:41.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.464+0000 7f645b9e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.464+0000 7f645b9e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.464+0000 7f645b9e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:41.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.464+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f645414f2d0 con 0x7f64541430c0 2024-03-25T19:45:41.490 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.490+0000 7f6449ffb640 1 -- 172.21.15.110:0/2106985209 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f644c052420 con 0x7f64541430c0 2024-03-25T19:45:41.490 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.490+0000 7f6449ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.490 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.490+0000 7f6449ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:41.490 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:45:41.491 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.493+0000 7f645b9e6640 10 librados: watch_flush enter 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.493+0000 7f645b9e6640 10 librados: watch_flush exit 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.493+0000 7f645b9e6640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.493+0000 7f645b9e6640 10 client.4455.objecter successfully canceled tick 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 >> v1:172.21.15.178:6800/1132739058 conn(0x7f643806e0c0 legacy=0x7f6438070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 10 monclient: shutdown 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 >> v1:172.21.15.110:6789/0 conn(0x7f64541430c0 legacy=0x7f64541e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 shutdown_connections 2024-03-25T19:45:41.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 >> 172.21.15.110:0/2106985209 conn(0x7f64540b7ea0 msgr2=0x7f6454074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:41.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 shutdown_connections 2024-03-25T19:45:41.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 1 -- 172.21.15.110:0/2106985209 wait complete. 2024-03-25T19:45:41.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:41.494+0000 7f645b9e6640 1 librados: shutdown 2024-03-25T19:45:41.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:42.952 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:50.236 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.002 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:50.286 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.025 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:50.310 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.049 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:50.303 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.055 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:50.309 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.074 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:50.358 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.088 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:50.372 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.102 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:05:50.357 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:43.113 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:50.398 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:48.312 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.424+0000 7fe4bfa7d640 10 monclient: get_monmap_and_config 2024-03-25T19:45:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.424+0000 7fe4bfa7d640 10 monclient: build_initial_monmap 2024-03-25T19:45:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.424+0000 7fe4bfa7d640 10 monclient: monmap: 2024-03-25T19:45:48.425 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:48.425 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:48.426 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:45:48.426766+0000 2024-03-25T19:45:48.426 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:45:48.426766+0000 2024-03-25T19:45:48.426 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:48.426 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:48.426 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:48.426 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:45:48.426 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:45:48.426 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.425+0000 7fe4bfa7d640 1 Processor -- start 2024-03-25T19:45:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.425+0000 7fe4bfa7d640 1 -- start start 2024-03-25T19:45:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.425+0000 7fe4bfa7d640 10 monclient: init 2024-03-25T19:45:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b80b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b80b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b814be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80af940 con 0x7fe4b80b1730 2024-03-25T19:45:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b0a70 con 0x7fe4b80b4d60 2024-03-25T19:45:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b0d50 con 0x7fe4b814be90 2024-03-25T19:45:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.426+0000 7fe4bfa7d640 10 monclient(hunting): authenticate will time out at 4519.568482s 2024-03-25T19:45:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.435+0000 7fe4bd819640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fe4b80b4d60 0x7fe4b813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:33248/0 (socket says 172.21.15.110:33248) 2024-03-25T19:45:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.435+0000 7fe4bd819640 1 -- 172.21.15.110:0/2289420505 learned_addr learned my addr 172.21.15.110:0/2289420505 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.456+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688652618 0 0) 0x7fe4b80b0a70 con 0x7fe4b80b4d60 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.456+0000 7fe4bc817640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.456+0000 7fe4bc817640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.456+0000 7fe4bc817640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.456+0000 7fe4bc817640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.456+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4003520 con 0x7fe4b80b4d60 2024-03-25T19:45:48.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.460+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730709306 0 0) 0x7fe4b80b0d50 con 0x7fe4b814be90 2024-03-25T19:45:48.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.460+0000 7fe4bc817640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.460+0000 7fe4bc817640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.460+0000 7fe4bc817640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.460+0000 7fe4bc817640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:45:48.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.460+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80b0a70 con 0x7fe4b814be90 2024-03-25T19:45:48.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.465+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628369939 0 0) 0x7fe4b80af940 con 0x7fe4b80b1730 2024-03-25T19:45:48.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.465+0000 7fe4bc817640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.465+0000 7fe4bc817640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.465+0000 7fe4bc817640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.465+0000 7fe4bc817640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:45:48.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.465+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80b0d50 con 0x7fe4b80b1730 2024-03-25T19:45:48.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3125272641 0 0) 0x7fe4a4003520 con 0x7fe4b80b4d60 2024-03-25T19:45:48.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 >> v1:172.21.15.110:6790/0 conn(0x7fe4b814be90 legacy=0x7fe4b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 >> v1:172.21.15.110:6789/0 conn(0x7fe4b80b1730 legacy=0x7fe4b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 10 monclient: _finish_hunting 0 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 1 monclient: found mon.noname-c 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80b1030 con 0x7fe4b80b4d60 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 10 monclient: _finish_auth 0 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 10 monclient: _check_auth_tickets 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe4b4001690 con 0x7fe4b80b4d60 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.468+0000 7fe4bc817640 10 monclient: dump: 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:48.470 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.469+0000 7fe4bfa7d640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:45:48.471 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.469+0000 7fe4bfa7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:48.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.479+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b40017f0 con 0x7fe4b80b4d60 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.479+0000 7fe4bc817640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.479+0000 7fe4bc817640 1 -- 172.21.15.110:0/2289420505 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe4b4001c00 con 0x7fe4b80b4d60 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.479+0000 7fe4bc817640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.479+0000 7fe4bc817640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.479+0000 7fe4bc817640 10 monclient: dump: 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:48.480 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:48.481 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:48.481 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:48.481 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.479+0000 7fe4bfa7d640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:48.481 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.479+0000 7fe4bfa7d640 4 set_mon_vals no callback set 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.480+0000 7fe4bfa7d640 10 monclient: shutdown 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.480+0000 7fe4bfa7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.480+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/2289420505 >> v1:172.21.15.178:6789/0 conn(0x7fe4b80b4d60 legacy=0x7fe4b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.480+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/2289420505 shutdown_connections 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.480+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/2289420505 >> 172.21.15.110:0/2289420505 conn(0x7fe4b80617a0 msgr2=0x7fe4b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.480+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/2289420505 shutdown_connections 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.480+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/2289420505 wait complete. 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.481+0000 7fe4bfa7d640 10 monclient: build_initial_monmap 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.481+0000 7fe4bfa7d640 10 monclient: monmap: 2024-03-25T19:45:48.482 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:48.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 librados: starting msgr at 2024-03-25T19:45:48.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 librados: starting objecter 2024-03-25T19:45:48.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 Processor -- start 2024-03-25T19:45:48.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 -- start start 2024-03-25T19:45:48.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 librados: setting wanted keys 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 librados: calling monclient init 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 10 monclient: init 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b80b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b80b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b814be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8141ec0 con 0x7fe4b814be90 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8142110 con 0x7fe4b80b4d60 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b81d79d0 con 0x7fe4b80b1730 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:48.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.482+0000 7fe4bfa7d640 10 monclient(hunting): authenticate will time out at 4519.624722s 2024-03-25T19:45:48.502 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.501+0000 7fe4bd018640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fe4b80b4d60 0x7fe4b81d7ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:33292/0 (socket says 172.21.15.110:33292) 2024-03-25T19:45:48.502 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.501+0000 7fe4bd018640 1 -- 172.21.15.110:0/67008845 learned_addr learned my addr 172.21.15.110:0/67008845 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:48.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.504+0000 7fe4ae7fc640 10 client.?.objecter ms_handle_connect 0x7fe4b80b4d60 2024-03-25T19:45:48.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.504+0000 7fe4ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:48.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.504+0000 7fe4ae7fc640 10 client.?.objecter ms_handle_connect 0x7fe4b814be90 2024-03-25T19:45:48.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.504+0000 7fe4ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:48.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.508+0000 7fe4ae7fc640 10 client.?.objecter ms_handle_connect 0x7fe4b80b1730 2024-03-25T19:45:48.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.508+0000 7fe4ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:48.514 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.514+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702662681 0 0) 0x7fe4b8142110 con 0x7fe4b80b4d60 2024-03-25T19:45:48.515 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.514+0000 7fe4ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.515 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.514+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.515 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.514+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.515 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.514+0000 7fe4ae7fc640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:45:48.515 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.514+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c004eb0 con 0x7fe4b80b4d60 2024-03-25T19:45:48.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731783413 0 0) 0x7fe4b8141ec0 con 0x7fe4b814be90 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8142110 con 0x7fe4b814be90 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539316611 0 0) 0x7fe4b81d79d0 con 0x7fe4b80b1730 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:45:48.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.519+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8141ec0 con 0x7fe4b80b1730 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.532+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1332453699 0 0) 0x7fe4b8141ec0 con 0x7fe4b80b1730 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.532+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b81d79d0 con 0x7fe4b80b1730 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.532+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe4b4001b00 con 0x7fe4b80b1730 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.532+0000 7fe4ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.532+0000 7fe4ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.532+0000 7fe4ae7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.533 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:48.534 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:48.534 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:48.534 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:48.534 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:48.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.532+0000 7fe4ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:48.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.534+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4068302086 0 0) 0x7fe49c004eb0 con 0x7fe4b80b4d60 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.534+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b8141ec0 con 0x7fe4b80b4d60 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.534+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe4a8001a10 con 0x7fe4b80b4d60 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.534+0000 7fe4ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.534+0000 7fe4ae7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.534+0000 7fe4ae7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:48.535 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:48.536 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:48.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4035950072 0 0) 0x7fe4b81d79d0 con 0x7fe4b80b1730 2024-03-25T19:45:48.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 >> v1:172.21.15.178:6789/0 conn(0x7fe4b80b4d60 legacy=0x7fe4b81d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 >> v1:172.21.15.110:6789/0 conn(0x7fe4b814be90 legacy=0x7fe4b8142d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:48.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 1 monclient: found mon.c 2024-03-25T19:45:48.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:48.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b81d8600 con 0x7fe4b80b1730 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 5 monclient: authenticate success, global_id 4469 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 10 monclient: _renew_subs 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b8143450 con 0x7fe4b80b1730 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 10 monclient: _renew_subs 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b813bb10 con 0x7fe4b80b1730 2024-03-25T19:45:48.553 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.551+0000 7fe4bfa7d640 1 librados: init done 2024-03-25T19:45:48.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.552+0000 7fe4bfa7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:48.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.552+0000 7fe4bfa7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:48.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.552+0000 7fe4bfa7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:45:48.554 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.552+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b81d7b60 con 0x7fe4b80b1730 2024-03-25T19:45:48.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.872+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b4001c60 con 0x7fe4b80b1730 2024-03-25T19:45:48.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.872+0000 7fe4ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:48.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.872+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe4b4004c30 con 0x7fe4b80b1730 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.872+0000 7fe4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.872+0000 7fe4ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.872+0000 7fe4ae7fc640 10 monclient: dump: 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:48.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.872+0000 7fe4affff640 4 set_mon_vals no callback set 2024-03-25T19:45:48.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.873+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fe4b401ac00 con 0x7fe4b80b1730 2024-03-25T19:45:48.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.874+0000 7fe4bd018640 10 monclient: get_auth_request con 0x7fe49c06df20 auth_method 0 2024-03-25T19:45:48.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.874+0000 7fe4ae7fc640 10 client.4469.objecter ms_handle_connect 0x7fe49c06df20 2024-03-25T19:45:48.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.880+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fe4b40010a0 con 0x7fe4b80b1730 2024-03-25T19:45:48.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.880+0000 7fe4ae7fc640 10 client.4469.objecter ms_dispatch 0x7fe4b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:48.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.880+0000 7fe4ae7fc640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:48.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.880+0000 7fe4ae7fc640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:48.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.880+0000 7fe4ae7fc640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-25T19:45:48.882 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.880+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4b40500c0 con 0x7fe4b80b1730 2024-03-25T19:45:48.882 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.880+0000 7fe4ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:48.882 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:48.880+0000 7fe4ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:49.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.139+0000 7fe4bfa7d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.139+0000 7fe4bfa7d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.139+0000 7fe4bfa7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:45:49.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.139+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b80627b0 con 0x7fe4b80b1730 2024-03-25T19:45:49.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.171+0000 7fe4ae7fc640 1 -- 172.21.15.110:0/67008845 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fe4b40521c0 con 0x7fe4b80b1730 2024-03-25T19:45:49.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.171+0000 7fe4ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.171+0000 7fe4ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:49.172 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:45:49.172 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:49.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 10 librados: watch_flush enter 2024-03-25T19:45:49.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 10 librados: watch_flush exit 2024-03-25T19:45:49.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-25T19:45:49.174 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 10 client.4469.objecter successfully canceled tick 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 >> v1:172.21.15.178:6800/1132739058 conn(0x7fe49c06df20 legacy=0x7fe49c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 10 monclient: shutdown 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 >> v1:172.21.15.110:6790/0 conn(0x7fe4b80b1730 legacy=0x7fe4b8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 shutdown_connections 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 >> 172.21.15.110:0/67008845 conn(0x7fe4b80617a0 msgr2=0x7fe4b80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 shutdown_connections 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 1 -- 172.21.15.110:0/67008845 wait complete. 2024-03-25T19:45:49.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:49.173+0000 7fe4bfa7d640 1 librados: shutdown 2024-03-25T19:45:49.186 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:55.992 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:56.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.104+0000 7f3c030c9640 10 monclient: get_monmap_and_config 2024-03-25T19:45:56.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.104+0000 7f3c030c9640 10 monclient: build_initial_monmap 2024-03-25T19:45:56.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.104+0000 7f3c030c9640 10 monclient: monmap: 2024-03-25T19:45:56.104 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:56.104 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:56.104 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:45:56.105402+0000 2024-03-25T19:45:56.104 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:45:56.105402+0000 2024-03-25T19:45:56.104 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:56.105 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:56.105 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:56.105 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:45:56.105 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:45:56.105 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.105+0000 7f3c030c9640 1 Processor -- start 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.105+0000 7f3c030c9640 1 -- start start 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.105+0000 7f3c030c9640 10 monclient: init 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc0705b0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 10 monclient(hunting): picked mon.noname-a con 0x7f3bfc1402d0 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 10 monclient(hunting): picked mon.noname-b con 0x7f3bfc149ae0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:56.106 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0718c0 con 0x7f3bfc1402d0 2024-03-25T19:45:56.107 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0714c0 con 0x7f3bfc0705b0 2024-03-25T19:45:56.107 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc072b50 con 0x7f3bfc149ae0 2024-03-25T19:45:56.107 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:56.107 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.106+0000 7f3c030c9640 10 monclient(hunting): authenticate will time out at 4527.247053s 2024-03-25T19:45:56.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.115+0000 7f3c00e65640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f3bfc0705b0 0x7f3bfc13d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:52390/0 (socket says 172.21.15.110:52390) 2024-03-25T19:45:56.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.115+0000 7f3c00e65640 1 -- 172.21.15.110:0/3907303923 learned_addr learned my addr 172.21.15.110:0/3907303923 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:56.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.134+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438713489 0 0) 0x7f3bfc0714c0 con 0x7f3bfc0705b0 2024-03-25T19:45:56.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.134+0000 7f3bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.134+0000 7f3bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.134+0000 7f3bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.134+0000 7f3bf37fe640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:45:56.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.134+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be4003520 con 0x7f3bfc0705b0 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.143+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827750814 0 0) 0x7f3bfc072b50 con 0x7f3bfc149ae0 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc0714c0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281586874 0 0) 0x7f3bfc0718c0 con 0x7f3bfc1402d0 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:45:56.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.144+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc072b50 con 0x7f3bfc1402d0 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124838996 0 0) 0x7f3be4003520 con 0x7f3bfc0705b0 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 >> v1:172.21.15.110:6790/0 conn(0x7f3bfc149ae0 legacy=0x7f3bfc14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 >> v1:172.21.15.110:6789/0 conn(0x7f3bfc1402d0 legacy=0x7f3bfc140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 1 monclient: found mon.noname-c 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc13dd10 con 0x7f3bfc0705b0 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:56.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3bf8001690 con 0x7f3bfc0705b0 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3bf37fe640 10 monclient: dump: 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:56.149 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:56.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3c030c9640 5 monclient: authenticate success, global_id 4474 2024-03-25T19:45:56.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.148+0000 7f3c030c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:56.158 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.158+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bf80017f0 con 0x7f3bfc0705b0 2024-03-25T19:45:56.158 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3bf37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:56.158 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3bf37fe640 1 -- 172.21.15.110:0/3907303923 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3bf8001c00 con 0x7f3bfc0705b0 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3c030c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3c030c9640 4 set_mon_vals no callback set 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3bf37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3bf37fe640 10 monclient: dump: 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:56.159 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:56.160 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3c030c9640 10 monclient: shutdown 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3c030c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3c030c9640 1 -- 172.21.15.110:0/3907303923 >> v1:172.21.15.178:6789/0 conn(0x7f3bfc0705b0 legacy=0x7f3bfc13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3c030c9640 1 -- 172.21.15.110:0/3907303923 shutdown_connections 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.159+0000 7f3c030c9640 1 -- 172.21.15.110:0/3907303923 >> 172.21.15.110:0/3907303923 conn(0x7f3bfc0b5d10 msgr2=0x7f3bfc0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.160+0000 7f3c030c9640 1 -- 172.21.15.110:0/3907303923 shutdown_connections 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.160+0000 7f3c030c9640 1 -- 172.21.15.110:0/3907303923 wait complete. 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.160+0000 7f3c030c9640 10 monclient: build_initial_monmap 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.160+0000 7f3c030c9640 10 monclient: monmap: 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:56.162 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.161+0000 7f3c030c9640 1 librados: starting msgr at 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.161+0000 7f3c030c9640 1 librados: starting objecter 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.161+0000 7f3c030c9640 1 Processor -- start 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.161+0000 7f3c030c9640 1 -- start start 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.161+0000 7f3c030c9640 1 librados: setting wanted keys 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.161+0000 7f3c030c9640 1 librados: calling monclient init 2024-03-25T19:45:56.163 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.161+0000 7f3c030c9640 10 monclient: init 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc0705b0 addr v1:172.21.15.110:6790/0 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 10 monclient(hunting): picked mon.noname-b con 0x7f3bfc1402d0 addr v1:172.21.15.178:6789/0 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 10 monclient(hunting): picked mon.noname-a con 0x7f3bfc149ae0 addr v1:172.21.15.110:6789/0 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc1439c0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc143ca0 con 0x7f3bfc1402d0 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc142c80 con 0x7f3bfc0705b0 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:56.164 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.162+0000 7f3c030c9640 10 monclient(hunting): authenticate will time out at 4527.303244s 2024-03-25T19:45:56.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.176+0000 7f3bf3fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f3bfc1402d0 0x7f3bfc1d57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:52426/0 (socket says 172.21.15.110:52426) 2024-03-25T19:45:56.175 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.176+0000 7f3bf3fff640 1 -- 172.21.15.110:0/2450540096 learned_addr learned my addr 172.21.15.110:0/2450540096 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:45:56.177 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.177+0000 7f3bf17fa640 10 client.?.objecter ms_handle_connect 0x7f3bfc1402d0 2024-03-25T19:45:56.177 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.177+0000 7f3bf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.183+0000 7f3bf17fa640 10 client.?.objecter ms_handle_connect 0x7f3bfc0705b0 2024-03-25T19:45:56.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.183+0000 7f3bf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.184+0000 7f3bf17fa640 10 client.?.objecter ms_handle_connect 0x7f3bfc149ae0 2024-03-25T19:45:56.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.184+0000 7f3bf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.188+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684107634 0 0) 0x7f3bfc143ca0 con 0x7f3bfc1402d0 2024-03-25T19:45:56.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.188+0000 7f3bf17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.188+0000 7f3bf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.188+0000 7f3bf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.188+0000 7f3bf17fa640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:45:56.188 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.188+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be8004eb0 con 0x7f3bfc1402d0 2024-03-25T19:45:56.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.189+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613312839 0 0) 0x7f3bfc1439c0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.189+0000 7f3bf17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.189+0000 7f3bf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.189+0000 7f3bf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.189+0000 7f3bf17fa640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:45:56.189 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.189+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc143ca0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.191+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991093583 0 0) 0x7f3bfc142c80 con 0x7f3bfc0705b0 2024-03-25T19:45:56.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.191+0000 7f3bf17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.191+0000 7f3bf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.191+0000 7f3bf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.191+0000 7f3bf17fa640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:45:56.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.191+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc1439c0 con 0x7f3bfc0705b0 2024-03-25T19:45:56.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.201+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 655357868 0 0) 0x7f3bfc143ca0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.202+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bfc142c80 con 0x7f3bfc149ae0 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.202+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3bf4001b50 con 0x7f3bfc149ae0 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.202+0000 7f3bf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.202+0000 7f3bf17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.202+0000 7f3bf17fa640 10 monclient(hunting): dump: 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:56.202 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:56.203 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:56.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.202+0000 7f3bf17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.206+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3715846019 0 0) 0x7f3be8004eb0 con 0x7f3bfc1402d0 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.206+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bfc143ca0 con 0x7f3bfc1402d0 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.206+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3bec001a10 con 0x7f3bfc1402d0 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.206+0000 7f3bf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.206+0000 7f3bf17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.206+0000 7f3bf17fa640 10 monclient(hunting): dump: 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:56.206 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:56.207 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:56.207 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:56.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4029909858 0 0) 0x7f3bfc142c80 con 0x7f3bfc149ae0 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 >> v1:172.21.15.110:6790/0 conn(0x7f3bfc0705b0 legacy=0x7f3bfc142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 >> v1:172.21.15.178:6789/0 conn(0x7f3bfc1402d0 legacy=0x7f3bfc1d57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 1 monclient: found mon.a 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc1429a0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3bf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 5 monclient: authenticate success, global_id 4467 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 10 monclient: _renew_subs 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc1443a0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:56.209 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 10 monclient: _renew_subs 2024-03-25T19:45:56.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:56.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc1e1190 con 0x7f3bfc149ae0 2024-03-25T19:45:56.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.208+0000 7f3c030c9640 1 librados: init done 2024-03-25T19:45:56.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.210+0000 7f3c030c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:56.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.210+0000 7f3c030c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:56.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.210+0000 7f3c030c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:56.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.210+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc0627b0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.220+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bf4002080 con 0x7f3bfc149ae0 2024-03-25T19:45:56.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.220+0000 7f3bf17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:56.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.220+0000 7f3bf2ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:56.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.523+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3bf4003b10 con 0x7f3bfc149ae0 2024-03-25T19:45:56.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.523+0000 7f3bf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.523+0000 7f3bf17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:56.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.523+0000 7f3bf17fa640 10 monclient: dump: 2024-03-25T19:45:56.523 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:45:56.523 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:45:56.524 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.524 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:45:56.524 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.524 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:45:56.524 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:45:56.524 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:45:56.524 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:45:56.524 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:45:56.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.524+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f3bf4019ae0 con 0x7f3bfc149ae0 2024-03-25T19:45:56.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.525+0000 7f3c00e65640 10 monclient: get_auth_request con 0x7f3be806df70 auth_method 0 2024-03-25T19:45:56.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.525+0000 7f3bf17fa640 10 client.4467.objecter ms_handle_connect 0x7f3be806df70 2024-03-25T19:45:56.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.530+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f3bf4053810 con 0x7f3bfc149ae0 2024-03-25T19:45:56.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.530+0000 7f3bf17fa640 10 client.4467.objecter ms_dispatch 0x7f3bfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:56.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.530+0000 7f3bf17fa640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:56.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.530+0000 7f3bf17fa640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:56.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.530+0000 7f3bf17fa640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-25T19:45:56.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.530+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bf404f150 con 0x7f3bfc149ae0 2024-03-25T19:45:56.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.530+0000 7f3bf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:56.530 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.530+0000 7f3bf17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:56.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.775+0000 7f3c030c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:56.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.775+0000 7f3c030c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:56.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.775+0000 7f3c030c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:45:56.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.775+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bfc0b5a10 con 0x7f3bfc149ae0 2024-03-25T19:45:56.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.794+0000 7f3bf17fa640 1 -- 172.21.15.110:0/2450540096 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f3bf4051250 con 0x7f3bfc149ae0 2024-03-25T19:45:56.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.794+0000 7f3bf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:56.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.794+0000 7f3bf17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:56.794 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:45:56.794 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.797+0000 7f3c030c9640 10 librados: watch_flush enter 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.797+0000 7f3c030c9640 10 librados: watch_flush exit 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.797+0000 7f3c030c9640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.797+0000 7f3c030c9640 10 client.4467.objecter successfully canceled tick 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.797+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 >> v1:172.21.15.178:6800/1132739058 conn(0x7f3be806df70 legacy=0x7f3be8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.797+0000 7f3c030c9640 10 monclient: shutdown 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.797+0000 7f3c030c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.797+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 >> v1:172.21.15.110:6789/0 conn(0x7f3bfc149ae0 legacy=0x7f3bfc1d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.798+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 shutdown_connections 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.798+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 >> 172.21.15.110:0/2450540096 conn(0x7f3bfc0b5d10 msgr2=0x7f3bfc13c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.798+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 shutdown_connections 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.798+0000 7f3c030c9640 1 -- 172.21.15.110:0/2450540096 wait complete. 2024-03-25T19:45:56.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:45:56.798+0000 7f3c030c9640 1 librados: shutdown 2024-03-25T19:45:56.808 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:03.035 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:10.319 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.040 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:10.325 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.059 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:10.343 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.095 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:10.379 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.095 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:10.349 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.098 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:10.353 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.105 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:10.390 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.131 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:10.386 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.137 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:10.421 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:03.615 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.729+0000 7fad48855640 10 monclient: get_monmap_and_config 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.729+0000 7fad48855640 10 monclient: build_initial_monmap 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.729+0000 7fad48855640 10 monclient: monmap: 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:46:03.730284+0000 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:46:03.730284+0000 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:03.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.730+0000 7fad48855640 1 Processor -- start 2024-03-25T19:46:03.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.730+0000 7fad48855640 1 -- start start 2024-03-25T19:46:03.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.730+0000 7fad48855640 10 monclient: init 2024-03-25T19:46:03.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.730+0000 7fad48855640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:03.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.730+0000 7fad48855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:03.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.730+0000 7fad48855640 10 monclient(hunting): picked mon.noname-c con 0x7fad440a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:03.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.730+0000 7fad48855640 10 monclient(hunting): picked mon.noname-a con 0x7fad4413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:03.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.731+0000 7fad48855640 10 monclient(hunting): picked mon.noname-b con 0x7fad441430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:03.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.731+0000 7fad48855640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad44142520 con 0x7fad4413f710 2024-03-25T19:46:03.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.731+0000 7fad48855640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad440b28a0 con 0x7fad440a98f0 2024-03-25T19:46:03.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.731+0000 7fad48855640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad440b2a30 con 0x7fad441430c0 2024-03-25T19:46:03.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.731+0000 7fad48855640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:03.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.731+0000 7fad48855640 10 monclient(hunting): authenticate will time out at 4534.871989s 2024-03-25T19:46:03.738 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.739+0000 7fad42d9d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fad441430c0 0x7fad4414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:45466/0 (socket says 172.21.15.110:45466) 2024-03-25T19:46:03.739 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.739+0000 7fad42d9d640 1 -- 172.21.15.110:0/1909780582 learned_addr learned my addr 172.21.15.110:0/1909780582 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:03.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.755+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886992696 0 0) 0x7fad440b2a30 con 0x7fad441430c0 2024-03-25T19:46:03.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.755+0000 7fad41d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.755+0000 7fad41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.755+0000 7fad41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.755+0000 7fad41d9b640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:46:03.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.755+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad30003520 con 0x7fad441430c0 2024-03-25T19:46:03.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.761+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206579549 0 0) 0x7fad440b28a0 con 0x7fad440a98f0 2024-03-25T19:46:03.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.761+0000 7fad41d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.761+0000 7fad41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.761+0000 7fad41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.761+0000 7fad41d9b640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:46:03.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.761+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad440b2a30 con 0x7fad440a98f0 2024-03-25T19:46:03.765 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.765+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674188706 0 0) 0x7fad44142520 con 0x7fad4413f710 2024-03-25T19:46:03.765 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.765+0000 7fad41d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.765 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.765+0000 7fad41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.765 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.765+0000 7fad41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.765 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.765+0000 7fad41d9b640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:46:03.765 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.765+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad440b28a0 con 0x7fad4413f710 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 95991945 0 0) 0x7fad440b2a30 con 0x7fad440a98f0 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 >> v1:172.21.15.110:6790/0 conn(0x7fad441430c0 legacy=0x7fad4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 >> v1:172.21.15.110:6789/0 conn(0x7fad4413f710 legacy=0x7fad441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 1 monclient: found mon.noname-c 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440b44c0 con 0x7fad440a98f0 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 10 monclient: _finish_auth 0 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:46:03.774 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fad34001670 con 0x7fad440a98f0 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad48855640 5 monclient: authenticate success, global_id 4480 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad48855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad41d9b640 10 monclient: dump: 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:03.775 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:03.776 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:03.776 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.774+0000 7fad48855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:03.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.789+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad340017d0 con 0x7fad440a98f0 2024-03-25T19:46:03.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.789+0000 7fad41d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:03.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.789+0000 7fad41d9b640 1 -- 172.21.15.110:0/1909780582 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fad34001be0 con 0x7fad440a98f0 2024-03-25T19:46:03.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.789+0000 7fad48855640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:03.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.789+0000 7fad48855640 4 set_mon_vals no callback set 2024-03-25T19:46:03.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.789+0000 7fad41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.789+0000 7fad41d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:03.789 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.789+0000 7fad41d9b640 10 monclient: dump: 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:03.790 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:03.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.790+0000 7fad48855640 10 monclient: shutdown 2024-03-25T19:46:03.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.790+0000 7fad48855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:03.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.790+0000 7fad48855640 1 -- 172.21.15.110:0/1909780582 >> v1:172.21.15.178:6789/0 conn(0x7fad440a98f0 legacy=0x7fad440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.790+0000 7fad48855640 1 -- 172.21.15.110:0/1909780582 shutdown_connections 2024-03-25T19:46:03.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.790+0000 7fad48855640 1 -- 172.21.15.110:0/1909780582 >> 172.21.15.110:0/1909780582 conn(0x7fad440b7ea0 msgr2=0x7fad440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:03.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.790+0000 7fad48855640 1 -- 172.21.15.110:0/1909780582 shutdown_connections 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.790+0000 7fad48855640 1 -- 172.21.15.110:0/1909780582 wait complete. 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.791+0000 7fad48855640 10 monclient: build_initial_monmap 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.791+0000 7fad48855640 10 monclient: monmap: 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 librados: starting msgr at 2024-03-25T19:46:03.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 librados: starting objecter 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 Processor -- start 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 -- start start 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 librados: setting wanted keys 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 librados: calling monclient init 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 10 monclient: init 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 10 monclient(hunting): picked mon.noname-b con 0x7fad440a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 10 monclient(hunting): picked mon.noname-c con 0x7fad4413f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 10 monclient(hunting): picked mon.noname-a con 0x7fad441430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad44139330 con 0x7fad441430c0 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad440abe70 con 0x7fad440a98f0 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad440ac150 con 0x7fad4413f710 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:03.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.792+0000 7fad48855640 10 monclient(hunting): authenticate will time out at 4534.933783s 2024-03-25T19:46:03.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.799+0000 7fad4259c640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fad440a98f0 0x7fad441389d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:45244/0 (socket says 172.21.15.110:45244) 2024-03-25T19:46:03.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.799+0000 7fad4259c640 1 -- 172.21.15.110:0/918854257 learned_addr learned my addr 172.21.15.110:0/918854257 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:03.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.802+0000 7fad3affd640 10 client.?.objecter ms_handle_connect 0x7fad440a98f0 2024-03-25T19:46:03.801 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.802+0000 7fad3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.805+0000 7fad3affd640 10 client.?.objecter ms_handle_connect 0x7fad441430c0 2024-03-25T19:46:03.804 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.805+0000 7fad3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.808+0000 7fad3affd640 10 client.?.objecter ms_handle_connect 0x7fad4413f710 2024-03-25T19:46:03.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.808+0000 7fad3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.813+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047758548 0 0) 0x7fad440ac150 con 0x7fad4413f710 2024-03-25T19:46:03.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.813+0000 7fad3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.813+0000 7fad3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.813+0000 7fad3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.813+0000 7fad3affd640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:46:03.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.813+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad24004eb0 con 0x7fad4413f710 2024-03-25T19:46:03.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.820+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066662369 0 0) 0x7fad440abe70 con 0x7fad440a98f0 2024-03-25T19:46:03.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.820+0000 7fad3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.820+0000 7fad3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.820+0000 7fad3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.820+0000 7fad3affd640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.820+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad440ac150 con 0x7fad440a98f0 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.821+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644853783 0 0) 0x7fad44139330 con 0x7fad441430c0 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.821+0000 7fad3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.821+0000 7fad3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.821+0000 7fad3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.821+0000 7fad3affd640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:46:03.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.821+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad440abe70 con 0x7fad441430c0 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.827+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1988202091 0 0) 0x7fad24004eb0 con 0x7fad4413f710 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.827+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad44139330 con 0x7fad4413f710 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.827+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fad2c0018f0 con 0x7fad4413f710 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.827+0000 7fad3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.827+0000 7fad3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.827+0000 7fad3affd640 10 monclient(hunting): dump: 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:03.827 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:03.828 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:03.828 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:03.828 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.827+0000 7fad3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:03.829 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.830+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2460615757 0 0) 0x7fad440ac150 con 0x7fad440a98f0 2024-03-25T19:46:03.829 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.830+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad24004eb0 con 0x7fad440a98f0 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.837+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fad34001670 con 0x7fad440a98f0 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.837+0000 7fad3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.837+0000 7fad3affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.837+0000 7fad3affd640 10 monclient(hunting): dump: 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:03.837 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:03.838 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.840+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1069743306 0 0) 0x7fad440abe70 con 0x7fad441430c0 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.840+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad440ac150 con 0x7fad441430c0 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.840+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fad3c001a30 con 0x7fad441430c0 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.840+0000 7fad3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.840+0000 7fad3affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.840+0000 7fad3affd640 10 monclient(hunting): dump: 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:03.840 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.841 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:03.841 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:03.841 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:03.841 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:03.841 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:03.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 281694129 0 0) 0x7fad24004eb0 con 0x7fad440a98f0 2024-03-25T19:46:03.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 >> v1:172.21.15.110:6790/0 conn(0x7fad4413f710 legacy=0x7fad440ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 >> v1:172.21.15.110:6789/0 conn(0x7fad441430c0 legacy=0x7fad441e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:03.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 1 monclient: found mon.b 2024-03-25T19:46:03.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440acc80 con 0x7fad440a98f0 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 10 monclient: _finish_auth 0 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 10 monclient: _renew_subs 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad44137e30 con 0x7fad440a98f0 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 10 monclient: _renew_subs 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad44136880 con 0x7fad440a98f0 2024-03-25T19:46:03.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.845+0000 7fad48855640 1 librados: init done 2024-03-25T19:46:03.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.846+0000 7fad48855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:03.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.846+0000 7fad48855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:03.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.846+0000 7fad48855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:03.847 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:03.846+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad44136b30 con 0x7fad440a98f0 2024-03-25T19:46:04.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.134+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad34003920 con 0x7fad440a98f0 2024-03-25T19:46:04.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.134+0000 7fad3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:04.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.134+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fad34003cf0 con 0x7fad440a98f0 2024-03-25T19:46:04.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.134+0000 7fad3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.134+0000 7fad3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:04.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.134+0000 7fad3affd640 10 monclient: dump: 2024-03-25T19:46:04.134 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:04.134 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:04.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.134+0000 7fad4159a640 4 set_mon_vals no callback set 2024-03-25T19:46:04.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.135+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fad34003f10 con 0x7fad440a98f0 2024-03-25T19:46:04.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.135+0000 7fad3bfff640 10 monclient: get_auth_request con 0x7fad2406e160 auth_method 0 2024-03-25T19:46:04.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.136+0000 7fad3affd640 10 client.4483.objecter ms_handle_connect 0x7fad2406e160 2024-03-25T19:46:04.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.141+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fad34054900 con 0x7fad440a98f0 2024-03-25T19:46:04.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.141+0000 7fad3affd640 10 client.4483.objecter ms_dispatch 0x7fad44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:04.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.141+0000 7fad3affd640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:04.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.141+0000 7fad3affd640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:04.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.141+0000 7fad3affd640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T19:46:04.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.141+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad34050410 con 0x7fad440a98f0 2024-03-25T19:46:04.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.141+0000 7fad3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:04.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.141+0000 7fad3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.363+0000 7fad48855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.363+0000 7fad48855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.363+0000 7fad48855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.363+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad440627b0 con 0x7fad440a98f0 2024-03-25T19:46:04.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.432+0000 7fad3affd640 1 -- 172.21.15.110:0/918854257 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fad34052340 con 0x7fad440a98f0 2024-03-25T19:46:04.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.432+0000 7fad3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:04.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.432+0000 7fad3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:04.432 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:46:04.433 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 10 librados: watch_flush enter 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 10 librados: watch_flush exit 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 10 client.4483.objecter successfully canceled tick 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 >> v1:172.21.15.178:6800/1132739058 conn(0x7fad2406e160 legacy=0x7fad24070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 10 monclient: shutdown 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 >> v1:172.21.15.178:6789/0 conn(0x7fad440a98f0 legacy=0x7fad441389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 shutdown_connections 2024-03-25T19:46:04.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 >> 172.21.15.110:0/918854257 conn(0x7fad440b7ea0 msgr2=0x7fad44074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:04.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 shutdown_connections 2024-03-25T19:46:04.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 1 -- 172.21.15.110:0/918854257 wait complete. 2024-03-25T19:46:04.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:04.435+0000 7fad48855640 1 librados: shutdown 2024-03-25T19:46:04.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:11.254 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:11.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.366+0000 7fa1bba25640 10 monclient: get_monmap_and_config 2024-03-25T19:46:11.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.366+0000 7fa1bba25640 10 monclient: build_initial_monmap 2024-03-25T19:46:11.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.366+0000 7fa1bba25640 10 monclient: monmap: 2024-03-25T19:46:11.366 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:11.366 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:11.366 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:46:11.367513+0000 2024-03-25T19:46:11.366 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:46:11.367513+0000 2024-03-25T19:46:11.367 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:11.367 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:11.367 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:11.367 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:46:11.367 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:46:11.367 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:11.369 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.367+0000 7fa1bba25640 1 Processor -- start 2024-03-25T19:46:11.369 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.367+0000 7fa1bba25640 1 -- start start 2024-03-25T19:46:11.369 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.367+0000 7fa1bba25640 10 monclient: init 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 10 monclient(hunting): picked mon.noname-c con 0x7fa1b40a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 10 monclient(hunting): picked mon.noname-a con 0x7fa1b413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 10 monclient(hunting): picked mon.noname-b con 0x7fa1b41430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b4142520 con 0x7fa1b413f710 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b40b28a0 con 0x7fa1b40a98f0 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b40b2a30 con 0x7fa1b41430c0 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:11.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.368+0000 7fa1bba25640 10 monclient(hunting): authenticate will time out at 4542.509104s 2024-03-25T19:46:11.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.372+0000 7fa1b8fc0640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa1b413f710 0x7fa1b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:50602/0 (socket says 172.21.15.110:50602) 2024-03-25T19:46:11.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.372+0000 7fa1b8fc0640 1 -- 172.21.15.110:0/2425666472 learned_addr learned my addr 172.21.15.110:0/2425666472 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:11.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.391+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706054853 0 0) 0x7fa1b4142520 con 0x7fa1b413f710 2024-03-25T19:46:11.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.392+0000 7fa1abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.392+0000 7fa1abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.392+0000 7fa1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.392+0000 7fa1abfff640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:46:11.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.392+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a0003520 con 0x7fa1b413f710 2024-03-25T19:46:11.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.396+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735155110 0 0) 0x7fa1b40b2a30 con 0x7fa1b41430c0 2024-03-25T19:46:11.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.396+0000 7fa1abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.396+0000 7fa1abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.396+0000 7fa1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.396+0000 7fa1abfff640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:46:11.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.396+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b4142520 con 0x7fa1b41430c0 2024-03-25T19:46:11.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.398+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116361254 0 0) 0x7fa1b40b28a0 con 0x7fa1b40a98f0 2024-03-25T19:46:11.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.398+0000 7fa1abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.398+0000 7fa1abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.398+0000 7fa1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.398+0000 7fa1abfff640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:46:11.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.398+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b40b2a30 con 0x7fa1b40a98f0 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220062179 0 0) 0x7fa1a0003520 con 0x7fa1b413f710 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 >> v1:172.21.15.110:6790/0 conn(0x7fa1b41430c0 legacy=0x7fa1b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 >> v1:172.21.15.178:6789/0 conn(0x7fa1b40a98f0 legacy=0x7fa1b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 1 monclient: found mon.noname-a 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b40b44c0 con 0x7fa1b413f710 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 10 monclient: _finish_auth 0 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa1a4001690 con 0x7fa1b413f710 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1abfff640 10 monclient: dump: 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1bba25640 5 monclient: authenticate success, global_id 4476 2024-03-25T19:46:11.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.402+0000 7fa1bba25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.407+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1a40017f0 con 0x7fa1b413f710 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.407+0000 7fa1abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.407+0000 7fa1abfff640 1 -- 172.21.15.110:0/2425666472 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa1a4001c00 con 0x7fa1b413f710 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.407+0000 7fa1abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.407+0000 7fa1abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.407+0000 7fa1abfff640 10 monclient: dump: 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.407 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:11.408 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:11.408 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:11.408 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:11.408 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:11.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.407+0000 7fa1bba25640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:11.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.407+0000 7fa1bba25640 4 set_mon_vals no callback set 2024-03-25T19:46:11.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.408+0000 7fa1bba25640 10 monclient: shutdown 2024-03-25T19:46:11.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.408+0000 7fa1bba25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:11.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.408+0000 7fa1bba25640 1 -- 172.21.15.110:0/2425666472 >> v1:172.21.15.110:6789/0 conn(0x7fa1b413f710 legacy=0x7fa1b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.408+0000 7fa1bba25640 1 -- 172.21.15.110:0/2425666472 shutdown_connections 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.408+0000 7fa1bba25640 1 -- 172.21.15.110:0/2425666472 >> 172.21.15.110:0/2425666472 conn(0x7fa1b40b7ea0 msgr2=0x7fa1b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.408+0000 7fa1bba25640 1 -- 172.21.15.110:0/2425666472 shutdown_connections 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.408+0000 7fa1bba25640 1 -- 172.21.15.110:0/2425666472 wait complete. 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.409+0000 7fa1bba25640 10 monclient: build_initial_monmap 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.409+0000 7fa1bba25640 10 monclient: monmap: 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:11.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.409+0000 7fa1bba25640 1 librados: starting msgr at 2024-03-25T19:46:11.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.409+0000 7fa1bba25640 1 librados: starting objecter 2024-03-25T19:46:11.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.409+0000 7fa1bba25640 1 Processor -- start 2024-03-25T19:46:11.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.409+0000 7fa1bba25640 1 -- start start 2024-03-25T19:46:11.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 1 librados: setting wanted keys 2024-03-25T19:46:11.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 1 librados: calling monclient init 2024-03-25T19:46:11.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 10 monclient: init 2024-03-25T19:46:11.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:11.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:11.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 10 monclient(hunting): picked mon.noname-a con 0x7fa1b40a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:11.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 10 monclient(hunting): picked mon.noname-b con 0x7fa1b413f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:11.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 10 monclient(hunting): picked mon.noname-c con 0x7fa1b41430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:11.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b41370e0 con 0x7fa1b40a98f0 2024-03-25T19:46:11.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b40b7260 con 0x7fa1b413f710 2024-03-25T19:46:11.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b40b73f0 con 0x7fa1b41430c0 2024-03-25T19:46:11.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:11.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.410+0000 7fa1bba25640 10 monclient(hunting): authenticate will time out at 4542.551622s 2024-03-25T19:46:11.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.415+0000 7fa1b9fc2640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fa1b41430c0 0x7fa1b41367d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:50270/0 (socket says 172.21.15.110:50270) 2024-03-25T19:46:11.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.415+0000 7fa1b9fc2640 1 -- 172.21.15.110:0/50604142 learned_addr learned my addr 172.21.15.110:0/50604142 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:11.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.418+0000 7fa1a9ffb640 10 client.?.objecter ms_handle_connect 0x7fa1b41430c0 2024-03-25T19:46:11.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.418+0000 7fa1a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:11.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.427+0000 7fa1a9ffb640 10 client.?.objecter ms_handle_connect 0x7fa1b413f710 2024-03-25T19:46:11.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.427+0000 7fa1a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:11.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.436+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895370872 0 0) 0x7fa1b40b73f0 con 0x7fa1b41430c0 2024-03-25T19:46:11.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.436+0000 7fa1a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.436+0000 7fa1a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.436+0000 7fa1a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.436+0000 7fa1a9ffb640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:46:11.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.436+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa198004eb0 con 0x7fa1b41430c0 2024-03-25T19:46:11.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.438+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925983261 0 0) 0x7fa1b40b7260 con 0x7fa1b413f710 2024-03-25T19:46:11.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.438+0000 7fa1a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.438+0000 7fa1a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.438+0000 7fa1a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.438+0000 7fa1a9ffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:46:11.438 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.438+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b40b73f0 con 0x7fa1b413f710 2024-03-25T19:46:11.443 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.444+0000 7fa1a9ffb640 10 client.?.objecter ms_handle_connect 0x7fa1b40a98f0 2024-03-25T19:46:11.443 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.444+0000 7fa1a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:11.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.453+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3206609792 0 0) 0x7fa1b40b73f0 con 0x7fa1b413f710 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.453+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1b40b7260 con 0x7fa1b413f710 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.453+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa1a4001b00 con 0x7fa1b413f710 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.453+0000 7fa1a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.453+0000 7fa1a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.453+0000 7fa1a9ffb640 10 monclient(hunting): dump: 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.454 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:11.455 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:11.455 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:11.455 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:11.455 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:11.455 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.453+0000 7fa1a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:11.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.458+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362733865 0 0) 0x7fa1b40b2a30 con 0x7fa1b40a98f0 2024-03-25T19:46:11.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.458+0000 7fa1a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.458+0000 7fa1a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.458+0000 7fa1a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.458+0000 7fa1a9ffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:46:11.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.458+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b40b73f0 con 0x7fa1b40a98f0 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1824146108 0 0) 0x7fa1b40b7260 con 0x7fa1b413f710 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 >> v1:172.21.15.110:6790/0 conn(0x7fa1b41430c0 legacy=0x7fa1b41367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 >> v1:172.21.15.110:6789/0 conn(0x7fa1b40a98f0 legacy=0x7fa1b4137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 1 monclient: found mon.b 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b4139630 con 0x7fa1b413f710 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1bba25640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:46:11.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1bba25640 10 monclient: _renew_subs 2024-03-25T19:46:11.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1bba25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:11.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.459+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b4139880 con 0x7fa1b413f710 2024-03-25T19:46:11.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.460+0000 7fa1bba25640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:11.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.460+0000 7fa1bba25640 10 monclient: _renew_subs 2024-03-25T19:46:11.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.460+0000 7fa1bba25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:11.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.460+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b41332e0 con 0x7fa1b413f710 2024-03-25T19:46:11.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.460+0000 7fa1bba25640 1 librados: init done 2024-03-25T19:46:11.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.461+0000 7fa1bba25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:11.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.461+0000 7fa1bba25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:11.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.461+0000 7fa1bba25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:11.462 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.461+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b4136ee0 con 0x7fa1b413f710 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.747+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1a4004a90 con 0x7fa1b413f710 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.747+0000 7fa1a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.747+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa1a4004da0 con 0x7fa1b413f710 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.747+0000 7fa1a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.747+0000 7fa1a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.747+0000 7fa1a9ffb640 10 monclient: dump: 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:11.747 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:11.748 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:11.748 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.748 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:11.748 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:11.748 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:11.748 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:11.748 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:11.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.747+0000 7fa1ab7fe640 4 set_mon_vals no callback set 2024-03-25T19:46:11.749 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.748+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fa1a401ad70 con 0x7fa1b413f710 2024-03-25T19:46:11.749 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.748+0000 7fa1b97c1640 10 monclient: get_auth_request con 0x7fa19806de10 auth_method 0 2024-03-25T19:46:11.749 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.749+0000 7fa1a9ffb640 10 client.4489.objecter ms_handle_connect 0x7fa19806de10 2024-03-25T19:46:11.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.754+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fa1a4054840 con 0x7fa1b413f710 2024-03-25T19:46:11.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.754+0000 7fa1a9ffb640 10 client.4489.objecter ms_dispatch 0x7fa1b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:11.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.754+0000 7fa1a9ffb640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:11.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.754+0000 7fa1a9ffb640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:11.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.754+0000 7fa1a9ffb640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T19:46:11.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.755+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1a4050350 con 0x7fa1b413f710 2024-03-25T19:46:11.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.755+0000 7fa1a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:11.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:11.755+0000 7fa1a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:12.008 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.008+0000 7fa1bba25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.008 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.008+0000 7fa1bba25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.008+0000 7fa1bba25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.008+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1b40627b0 con 0x7fa1b413f710 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.034+0000 7fa1a9ffb640 1 -- 172.21.15.110:0/50604142 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fa1a4052280 con 0x7fa1b413f710 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.034+0000 7fa1a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.034+0000 7fa1a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 10 librados: watch_flush enter 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 10 librados: watch_flush exit 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 10 client.4489.objecter successfully canceled tick 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 >> v1:172.21.15.178:6800/1132739058 conn(0x7fa19806de10 legacy=0x7fa1980702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 10 monclient: shutdown 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 >> v1:172.21.15.178:6789/0 conn(0x7fa1b413f710 legacy=0x7fa1b41daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 shutdown_connections 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 >> 172.21.15.110:0/50604142 conn(0x7fa1b40b7ea0 msgr2=0x7fa1b413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:12.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 shutdown_connections 2024-03-25T19:46:12.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 1 -- 172.21.15.110:0/50604142 wait complete. 2024-03-25T19:46:12.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:12.036+0000 7fa1bba25640 1 librados: shutdown 2024-03-25T19:46:12.048 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:18.854 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:18.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.972+0000 7f04ba1e8640 10 monclient: get_monmap_and_config 2024-03-25T19:46:18.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.972+0000 7f04ba1e8640 10 monclient: build_initial_monmap 2024-03-25T19:46:18.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.972+0000 7f04ba1e8640 10 monclient: monmap: 2024-03-25T19:46:18.972 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:18.972 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:18.972 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:46:18.973539+0000 2024-03-25T19:46:18.972 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:46:18.973539+0000 2024-03-25T19:46:18.972 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:18.973 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:18.973 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:18.973 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:46:18.973 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:46:18.973 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.973+0000 7f04ba1e8640 1 Processor -- start 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.973+0000 7f04ba1e8640 1 -- start start 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.973+0000 7f04ba1e8640 10 monclient: init 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.973+0000 7f04ba1e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.973+0000 7f04ba1e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.974+0000 7f04ba1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f04b40b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.974+0000 7f04ba1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f04b40b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.974+0000 7f04ba1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f04b414be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.974+0000 7f04ba1e8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40af940 con 0x7f04b40b4d60 2024-03-25T19:46:18.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.974+0000 7f04ba1e8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40b0a70 con 0x7f04b40b1730 2024-03-25T19:46:18.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.974+0000 7f04ba1e8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40b0d50 con 0x7f04b414be90 2024-03-25T19:46:18.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.974+0000 7f04ba1e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:18.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.974+0000 7f04ba1e8640 10 monclient(hunting): authenticate will time out at 4550.114981s 2024-03-25T19:46:18.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.979+0000 7f04b37fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f04b40b4d60 0x7f04b413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34814/0 (socket says 172.21.15.110:34814) 2024-03-25T19:46:18.978 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.979+0000 7f04b37fe640 1 -- 172.21.15.110:0/4153595764 learned_addr learned my addr 172.21.15.110:0/4153595764 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:18.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.987+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661864751 0 0) 0x7f04b40af940 con 0x7f04b40b4d60 2024-03-25T19:46:18.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.987+0000 7f04b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.987+0000 7f04b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.987+0000 7f04b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.987+0000 7f04b27fc640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:46:18.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.987+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c003520 con 0x7f04b40b4d60 2024-03-25T19:46:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.992+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813016315 0 0) 0x7f04b40b0d50 con 0x7f04b414be90 2024-03-25T19:46:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.992+0000 7f04b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.992+0000 7f04b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.992+0000 7f04b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.992+0000 7f04b27fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:46:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.992+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b40af940 con 0x7f04b414be90 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3641754906 0 0) 0x7f049c003520 con 0x7f04b40b4d60 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 >> v1:172.21.15.110:6790/0 conn(0x7f04b414be90 legacy=0x7f04b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 >> v1:172.21.15.178:6789/0 conn(0x7f04b40b1730 legacy=0x7f04b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 1 monclient: found mon.noname-a 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40b1030 con 0x7f04b40b4d60 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:18.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f04a4001690 con 0x7f04b40b4d60 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04b27fc640 10 monclient: dump: 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.997 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:18.998 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:18.998 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:18.998 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:18.998 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:18.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04ba1e8640 5 monclient: authenticate success, global_id 4482 2024-03-25T19:46:18.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:18.996+0000 7f04ba1e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:19.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.002+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04a40017f0 con 0x7f04b40b4d60 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.002+0000 7f04b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.002+0000 7f04b27fc640 1 -- 172.21.15.110:0/4153595764 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f04a4001c00 con 0x7f04b40b4d60 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.002+0000 7f04b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.002+0000 7f04b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.002+0000 7f04b27fc640 10 monclient: dump: 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:19.002 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:19.003 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:19.003 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.002+0000 7f04ba1e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:19.003 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.002+0000 7f04ba1e8640 4 set_mon_vals no callback set 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.003+0000 7f04ba1e8640 10 monclient: shutdown 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.003+0000 7f04ba1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.003+0000 7f04ba1e8640 1 -- 172.21.15.110:0/4153595764 >> v1:172.21.15.110:6789/0 conn(0x7f04b40b4d60 legacy=0x7f04b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.003+0000 7f04ba1e8640 1 -- 172.21.15.110:0/4153595764 shutdown_connections 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.003+0000 7f04ba1e8640 1 -- 172.21.15.110:0/4153595764 >> 172.21.15.110:0/4153595764 conn(0x7f04b40617a0 msgr2=0x7f04b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.003+0000 7f04ba1e8640 1 -- 172.21.15.110:0/4153595764 shutdown_connections 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.003+0000 7f04ba1e8640 1 -- 172.21.15.110:0/4153595764 wait complete. 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 10 monclient: build_initial_monmap 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 10 monclient: monmap: 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:46:19.005 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 1 librados: starting msgr at 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 1 librados: starting objecter 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 1 Processor -- start 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 1 -- start start 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 1 librados: setting wanted keys 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 1 librados: calling monclient init 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.004+0000 7f04ba1e8640 10 monclient: init 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f04b40b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f04b40b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f04b414be90 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b4138060 con 0x7f04b40b4d60 2024-03-25T19:46:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b41382d0 con 0x7f04b414be90 2024-03-25T19:46:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b41367c0 con 0x7f04b40b1730 2024-03-25T19:46:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.005+0000 7f04ba1e8640 10 monclient(hunting): authenticate will time out at 4550.146227s 2024-03-25T19:46:19.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.014+0000 7f04b37fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f04b40b1730 0x7f04b412fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:57686/0 (socket says 172.21.15.110:57686) 2024-03-25T19:46:19.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.014+0000 7f04b37fe640 1 -- 172.21.15.110:0/3178925669 learned_addr learned my addr 172.21.15.110:0/3178925669 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:19.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.017+0000 7f049bfff640 10 client.?.objecter ms_handle_connect 0x7f04b40b1730 2024-03-25T19:46:19.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.017+0000 7f049bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.026 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.026+0000 7f049bfff640 10 client.?.objecter ms_handle_connect 0x7f04b40b4d60 2024-03-25T19:46:19.026 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.026+0000 7f049bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.029+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642243261 0 0) 0x7f04b41367c0 con 0x7f04b40b1730 2024-03-25T19:46:19.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.029+0000 7f049bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.029+0000 7f049bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.029+0000 7f049bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.029+0000 7f049bfff640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:46:19.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.029+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a0004eb0 con 0x7f04b40b1730 2024-03-25T19:46:19.032 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.032+0000 7f049bfff640 10 client.?.objecter ms_handle_connect 0x7f04b414be90 2024-03-25T19:46:19.032 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.032+0000 7f049bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.041+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919864747 0 0) 0x7f04b4138060 con 0x7f04b40b4d60 2024-03-25T19:46:19.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.041+0000 7f049bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.041+0000 7f049bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.041 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.041+0000 7f049bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.041 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.041+0000 7f049bfff640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:46:19.041 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.041+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b41367c0 con 0x7f04b40b4d60 2024-03-25T19:46:19.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.049+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990033913 0 0) 0x7f04b41382d0 con 0x7f04b414be90 2024-03-25T19:46:19.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.049+0000 7f049bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.049+0000 7f049bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.049+0000 7f049bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.049+0000 7f049bfff640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:46:19.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.049+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b4138060 con 0x7f04b414be90 2024-03-25T19:46:19.050 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.050+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361410380 0 0) 0x7f04a0004eb0 con 0x7f04b40b1730 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.050+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04b41382d0 con 0x7f04b40b1730 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.050+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f04a4001bc0 con 0x7f04b40b1730 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.050+0000 7f049bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.050+0000 7f049bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.050+0000 7f049bfff640 10 monclient(hunting): dump: 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:19.051 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.052 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:19.052 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:19.052 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:19.052 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:19.052 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:19.052 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.050+0000 7f049bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.055+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653353147 0 0) 0x7f04b41367c0 con 0x7f04b40b4d60 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.055+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a0004eb0 con 0x7f04b40b4d60 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.055+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f04a80018b0 con 0x7f04b40b4d60 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.055+0000 7f049bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.055+0000 7f049bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.055+0000 7f049bfff640 10 monclient(hunting): dump: 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:19.055 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:19.056 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:19.056 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:19.056 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:19.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 682622466 0 0) 0x7f04b41382d0 con 0x7f04b40b1730 2024-03-25T19:46:19.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 >> v1:172.21.15.178:6789/0 conn(0x7f04b414be90 legacy=0x7f04b41391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 >> v1:172.21.15.110:6789/0 conn(0x7f04b40b4d60 legacy=0x7f04b4144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:19.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 1 monclient: found mon.c 2024-03-25T19:46:19.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:19.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4136ce0 con 0x7f04b40b1730 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 10 monclient: _finish_auth 0 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f049bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 5 monclient: authenticate success, global_id 4493 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 10 monclient: _renew_subs 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b4130470 con 0x7f04b40b1730 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 10 monclient: _renew_subs 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b41361c0 con 0x7f04b40b1730 2024-03-25T19:46:19.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.059+0000 7f04ba1e8640 1 librados: init done 2024-03-25T19:46:19.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.060+0000 7f04ba1e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:19.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.060+0000 7f04ba1e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:19.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.060+0000 7f04ba1e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:19.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.060+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b4136950 con 0x7f04b40b1730 2024-03-25T19:46:19.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.366+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04a40011f0 con 0x7f04b40b1730 2024-03-25T19:46:19.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.366+0000 7f049bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:19.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.366+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f04a4004cb0 con 0x7f04b40b1730 2024-03-25T19:46:19.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.366+0000 7f049bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.366+0000 7f049bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:19.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.366+0000 7f049bfff640 10 monclient: dump: 2024-03-25T19:46:19.366 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:19.366 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:19.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.366+0000 7f04b1ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:19.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.367+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f04a401ad00 con 0x7f04b40b1730 2024-03-25T19:46:19.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.367+0000 7f04b2ffd640 10 monclient: get_auth_request con 0x7f04a006ded0 auth_method 0 2024-03-25T19:46:19.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.368+0000 7f049bfff640 10 client.4493.objecter ms_handle_connect 0x7f04a006ded0 2024-03-25T19:46:19.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.373+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f04a40549f0 con 0x7f04b40b1730 2024-03-25T19:46:19.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.373+0000 7f049bfff640 10 client.4493.objecter ms_dispatch 0x7f04b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:19.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.373+0000 7f049bfff640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:19.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.373+0000 7f049bfff640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:19.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.373+0000 7f049bfff640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-25T19:46:19.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.374+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04a4050330 con 0x7f04b40b1730 2024-03-25T19:46:19.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.374+0000 7f049bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:19.374 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.374+0000 7f049bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:19.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.634+0000 7f04ba1e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:19.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.634+0000 7f04ba1e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:19.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.634+0000 7f04ba1e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:19.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.634+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04b4136e70 con 0x7f04b40b1730 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.667+0000 7f049bfff640 1 -- 172.21.15.110:0/3178925669 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f04a4052430 con 0x7f04b40b1730 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.667+0000 7f049bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.667+0000 7f049bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:19.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 10 librados: watch_flush enter 2024-03-25T19:46:19.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 10 librados: watch_flush exit 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 10 client.4493.objecter successfully canceled tick 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 >> v1:172.21.15.178:6800/1132739058 conn(0x7f04a006ded0 legacy=0x7f04a0070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 10 monclient: shutdown 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 >> v1:172.21.15.110:6790/0 conn(0x7f04b40b1730 legacy=0x7f04b412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 shutdown_connections 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 >> 172.21.15.110:0/3178925669 conn(0x7f04b40617a0 msgr2=0x7f04b413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 shutdown_connections 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.670+0000 7f04ba1e8640 1 -- 172.21.15.110:0/3178925669 wait complete. 2024-03-25T19:46:19.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:19.671+0000 7f04ba1e8640 1 librados: shutdown 2024-03-25T19:46:19.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:23.062 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:30.346 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.111 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:30.365 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.114 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:30.398 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.121 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:30.376 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.154 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:30.439 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.161 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:30.416 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.189 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:30.473 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.194 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:30.478 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.223 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:30.507 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.489 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:26.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.606+0000 7fb300f54640 10 monclient: get_monmap_and_config 2024-03-25T19:46:26.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.606+0000 7fb300f54640 10 monclient: build_initial_monmap 2024-03-25T19:46:26.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.606+0000 7fb300f54640 10 monclient: monmap: 2024-03-25T19:46:26.606 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:26.606 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:26.606 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:46:26.607454+0000 2024-03-25T19:46:26.606 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:46:26.607454+0000 2024-03-25T19:46:26.606 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:26.607 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:26.607 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:26.607 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:46:26.607 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:46:26.607 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:26.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.607+0000 7fb300f54640 1 Processor -- start 2024-03-25T19:46:26.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.607+0000 7fb300f54640 1 -- start start 2024-03-25T19:46:26.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.607+0000 7fb300f54640 10 monclient: init 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 10 monclient(hunting): picked mon.noname-c con 0x7fb2fc0b1ef0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 10 monclient(hunting): picked mon.noname-a con 0x7fb2fc1473a0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 10 monclient(hunting): picked mon.noname-b con 0x7fb2fc150070 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0b2570 con 0x7fb2fc1473a0 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc146c20 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc146e80 con 0x7fb2fc150070 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:26.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.608+0000 7fb300f54640 10 monclient(hunting): authenticate will time out at 4557.749375s 2024-03-25T19:46:26.610 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.611+0000 7fb2fa59c640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fb2fc0b1ef0 0x7fb2fc0b28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:45340/0 (socket says 172.21.15.110:45340) 2024-03-25T19:46:26.610 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.611+0000 7fb2fa59c640 1 -- 172.21.15.110:0/196710788 learned_addr learned my addr 172.21.15.110:0/196710788 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:26.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.690+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546557016 0 0) 0x7fb2fc0b2570 con 0x7fb2fc1473a0 2024-03-25T19:46:26.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.690+0000 7fb2f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:26.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.690+0000 7fb2f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:26.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.690+0000 7fb2f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:26.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.690+0000 7fb2f959a640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:46:26.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.690+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e4003520 con 0x7fb2fc1473a0 2024-03-25T19:46:26.694 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.694+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524938056 0 0) 0x7fb2fc146c20 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.694 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.694+0000 7fb2f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:26.694 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.694+0000 7fb2f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:26.694 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.694+0000 7fb2f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:26.694 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.694+0000 7fb2f959a640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:46:26.694 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.694+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc0b2570 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.697+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278080348 0 0) 0x7fb2fc146e80 con 0x7fb2fc150070 2024-03-25T19:46:26.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.697+0000 7fb2f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:26.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.697+0000 7fb2f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:26.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.697+0000 7fb2f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:26.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.697+0000 7fb2f959a640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:46:26.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.697+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc146c20 con 0x7fb2fc150070 2024-03-25T19:46:26.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347598465 0 0) 0x7fb2e4003520 con 0x7fb2fc1473a0 2024-03-25T19:46:26.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 >> v1:172.21.15.110:6790/0 conn(0x7fb2fc150070 legacy=0x7fb2fc152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 >> v1:172.21.15.178:6789/0 conn(0x7fb2fc0b1ef0 legacy=0x7fb2fc0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 10 monclient: _finish_hunting 0 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 1 monclient: found mon.noname-a 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc06e1b0 con 0x7fb2fc1473a0 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 10 monclient: _finish_auth 0 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 10 monclient: _check_auth_tickets 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb2f0001690 con 0x7fb2fc1473a0 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.703+0000 7fb2f959a640 10 monclient: dump: 2024-03-25T19:46:26.704 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.704+0000 7fb300f54640 5 monclient: authenticate success, global_id 4488 2024-03-25T19:46:26.705 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.704+0000 7fb300f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:26.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.720+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2f00017f0 con 0x7fb2fc1473a0 2024-03-25T19:46:26.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.720+0000 7fb2f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:26.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.720+0000 7fb2f959a640 1 -- 172.21.15.110:0/196710788 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb2f0001c00 con 0x7fb2fc1473a0 2024-03-25T19:46:26.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.720+0000 7fb300f54640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:26.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.720+0000 7fb300f54640 4 set_mon_vals no callback set 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.720+0000 7fb2f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.720+0000 7fb2f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.720+0000 7fb2f959a640 10 monclient: dump: 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:26.721 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:26.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.721+0000 7fb300f54640 10 monclient: shutdown 2024-03-25T19:46:26.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.721+0000 7fb300f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.721+0000 7fb300f54640 1 -- 172.21.15.110:0/196710788 >> v1:172.21.15.110:6789/0 conn(0x7fb2fc1473a0 legacy=0x7fb2fc143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.721+0000 7fb300f54640 1 -- 172.21.15.110:0/196710788 shutdown_connections 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.721+0000 7fb300f54640 1 -- 172.21.15.110:0/196710788 >> 172.21.15.110:0/196710788 conn(0x7fb2fc0b7ea0 msgr2=0x7fb2fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.721+0000 7fb300f54640 1 -- 172.21.15.110:0/196710788 shutdown_connections 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.721+0000 7fb300f54640 1 -- 172.21.15.110:0/196710788 wait complete. 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.722+0000 7fb300f54640 10 monclient: build_initial_monmap 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.722+0000 7fb300f54640 10 monclient: monmap: 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:26.724 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.722+0000 7fb300f54640 1 librados: starting msgr at 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.722+0000 7fb300f54640 1 librados: starting objecter 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 1 Processor -- start 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 1 -- start start 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 1 librados: setting wanted keys 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 1 librados: calling monclient init 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 10 monclient: init 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 10 monclient(hunting): picked mon.noname-a con 0x7fb2fc0b1ef0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 10 monclient(hunting): picked mon.noname-c con 0x7fb2fc1473a0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:26.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 10 monclient(hunting): picked mon.noname-b con 0x7fb2fc150070 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:26.726 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc1359f0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.726 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc071690 con 0x7fb2fc150070 2024-03-25T19:46:26.726 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc144790 con 0x7fb2fc1473a0 2024-03-25T19:46:26.726 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:26.726 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.723+0000 7fb300f54640 10 monclient(hunting): authenticate will time out at 4557.864568s 2024-03-25T19:46:26.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.729+0000 7fb2fa59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb2fc0b1ef0 0x7fb2fc134bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40290/0 (socket says 172.21.15.110:40290) 2024-03-25T19:46:26.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.729+0000 7fb2fa59c640 1 -- 172.21.15.110:0/3367151859 learned_addr learned my addr 172.21.15.110:0/3367151859 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:26.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.730+0000 7fb2eaffd640 10 client.?.objecter ms_handle_connect 0x7fb2fc1473a0 2024-03-25T19:46:26.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.730+0000 7fb2eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:26.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.730+0000 7fb2eaffd640 10 client.?.objecter ms_handle_connect 0x7fb2fc0b1ef0 2024-03-25T19:46:26.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.730+0000 7fb2eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:26.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.735+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365571487 0 0) 0x7fb2fc1359f0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.736+0000 7fb2eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:26.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.736+0000 7fb2eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:26.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.736+0000 7fb2eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:26.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.736+0000 7fb2eaffd640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:46:26.736 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.736+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2dc004eb0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.739 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.739+0000 7fb2eaffd640 10 client.?.objecter ms_handle_connect 0x7fb2fc150070 2024-03-25T19:46:26.739 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.739+0000 7fb2eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:26.743 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.743+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 800596427 0 0) 0x7fb2dc004eb0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.743 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.743+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2fc1359f0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.743 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.743+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb2f4001d70 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.743+0000 7fb2eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.743+0000 7fb2eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.743+0000 7fb2eaffd640 10 monclient(hunting): dump: 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:26.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.743+0000 7fb2eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:26.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.748+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963547556 0 0) 0x7fb2fc144790 con 0x7fb2fc1473a0 2024-03-25T19:46:26.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.748+0000 7fb2eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:26.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.748+0000 7fb2eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:26.747 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.748+0000 7fb2eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:26.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.748+0000 7fb2eaffd640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:46:26.748 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.748+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2dc004eb0 con 0x7fb2fc1473a0 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2540839802 0 0) 0x7fb2fc1359f0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 >> v1:172.21.15.110:6790/0 conn(0x7fb2fc1473a0 legacy=0x7fb2fc136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 >> v1:172.21.15.178:6789/0 conn(0x7fb2fc150070 legacy=0x7fb2fc1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 1 monclient: found mon.a 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc144a70 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:26.752 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb2eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 5 monclient: authenticate success, global_id 4491 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 10 monclient: _renew_subs 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc144e90 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 10 monclient: _renew_subs 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc070db0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:26.753 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.752+0000 7fb300f54640 1 librados: init done 2024-03-25T19:46:26.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.753+0000 7fb300f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:26.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.753+0000 7fb300f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:26.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.753+0000 7fb300f54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:26.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:26.753+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc0627b0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:27.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.043+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2f4002010 con 0x7fb2fc0b1ef0 2024-03-25T19:46:27.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.043+0000 7fb2eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:27.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.043+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fb2f4004fb0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:27.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.043+0000 7fb2eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.043+0000 7fb2eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.043+0000 7fb2eaffd640 10 monclient: dump: 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.043+0000 7fb2f8d99640 4 set_mon_vals no callback set 2024-03-25T19:46:27.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.044+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fb2f401af80 con 0x7fb2fc0b1ef0 2024-03-25T19:46:27.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.044+0000 7fb2f9d9b640 10 monclient: get_auth_request con 0x7fb2dc06dd90 auth_method 0 2024-03-25T19:46:27.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.045+0000 7fb2eaffd640 10 client.4491.objecter ms_handle_connect 0x7fb2dc06dd90 2024-03-25T19:46:27.050 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.051+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fb2f4054be0 con 0x7fb2fc0b1ef0 2024-03-25T19:46:27.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.051+0000 7fb2eaffd640 10 client.4491.objecter ms_dispatch 0x7fb2fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:27.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.051+0000 7fb2eaffd640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:27.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.051+0000 7fb2eaffd640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:27.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.051+0000 7fb2eaffd640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T19:46:27.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.051+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2f4050520 con 0x7fb2fc0b1ef0 2024-03-25T19:46:27.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.051+0000 7fb2eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:27.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.051+0000 7fb2eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:27.311 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.312+0000 7fb300f54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:27.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.312+0000 7fb300f54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:27.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.312+0000 7fb300f54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:27.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.312+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2fc142320 con 0x7fb2fc0b1ef0 2024-03-25T19:46:27.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.340+0000 7fb2eaffd640 1 -- 172.21.15.110:0/3367151859 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fb2f4052620 con 0x7fb2fc0b1ef0 2024-03-25T19:46:27.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.340+0000 7fb2eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:27.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.340+0000 7fb2eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:27.340 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:46:27.341 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:27.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 10 librados: watch_flush enter 2024-03-25T19:46:27.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 10 librados: watch_flush exit 2024-03-25T19:46:27.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T19:46:27.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 10 client.4491.objecter successfully canceled tick 2024-03-25T19:46:27.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 >> v1:172.21.15.178:6800/1132739058 conn(0x7fb2dc06dd90 legacy=0x7fb2dc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 10 monclient: shutdown 2024-03-25T19:46:27.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:27.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 >> v1:172.21.15.110:6789/0 conn(0x7fb2fc0b1ef0 legacy=0x7fb2fc134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 shutdown_connections 2024-03-25T19:46:27.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 >> 172.21.15.110:0/3367151859 conn(0x7fb2fc0b7ea0 msgr2=0x7fb2fc141810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:27.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 shutdown_connections 2024-03-25T19:46:27.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 1 -- 172.21.15.110:0/3367151859 wait complete. 2024-03-25T19:46:27.344 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:27.343+0000 7fb300f54640 1 librados: shutdown 2024-03-25T19:46:27.356 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:34.162 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.281+0000 7f63069e6640 10 monclient: get_monmap_and_config 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.281+0000 7f63069e6640 10 monclient: build_initial_monmap 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.281+0000 7f63069e6640 10 monclient: monmap: 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:46:34.282365+0000 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:46:34.282365+0000 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:34.281 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:34.282 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:46:34.282 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:46:34.282 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:34.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.282+0000 7f63069e6640 1 Processor -- start 2024-03-25T19:46:34.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.282+0000 7f63069e6640 1 -- start start 2024-03-25T19:46:34.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.282+0000 7f63069e6640 10 monclient: init 2024-03-25T19:46:34.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.282+0000 7f63069e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:34.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.282+0000 7f63069e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:34.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.282+0000 7f63069e6640 10 monclient(hunting): picked mon.noname-b con 0x7f63000a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:34.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.283+0000 7f63069e6640 10 monclient(hunting): picked mon.noname-a con 0x7f630013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:34.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.283+0000 7f63069e6640 10 monclient(hunting): picked mon.noname-c con 0x7f63001430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:34.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.283+0000 7f63069e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6300142520 con 0x7f630013f710 2024-03-25T19:46:34.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.283+0000 7f63069e6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63000b28a0 con 0x7f63001430c0 2024-03-25T19:46:34.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.283+0000 7f63069e6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63000b2a30 con 0x7f63000a98f0 2024-03-25T19:46:34.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.283+0000 7f63069e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:34.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.283+0000 7f63069e6640 10 monclient(hunting): authenticate will time out at 4565.423893s 2024-03-25T19:46:34.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.301+0000 7f62fffff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f63000a98f0 0x7f63000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:60088/0 (socket says 172.21.15.110:60088) 2024-03-25T19:46:34.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.301+0000 7f62fffff640 1 -- 172.21.15.110:0/914598558 learned_addr learned my addr 172.21.15.110:0/914598558 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:34.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.315+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197916812 0 0) 0x7f6300142520 con 0x7f630013f710 2024-03-25T19:46:34.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.315+0000 7f62feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.315+0000 7f62feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.315+0000 7f62feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.315+0000 7f62feffd640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:46:34.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.315+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e8003520 con 0x7f630013f710 2024-03-25T19:46:34.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.317+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013593658 0 0) 0x7f63000b2a30 con 0x7f63000a98f0 2024-03-25T19:46:34.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.317+0000 7f62feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.317+0000 7f62feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.317+0000 7f62feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.317+0000 7f62feffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:46:34.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.317+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6300142520 con 0x7f63000a98f0 2024-03-25T19:46:34.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.321+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566957585 0 0) 0x7f63000b28a0 con 0x7f63001430c0 2024-03-25T19:46:34.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.321+0000 7f62feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.321+0000 7f62feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.321+0000 7f62feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.321+0000 7f62feffd640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:46:34.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.321+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63000b2a30 con 0x7f63001430c0 2024-03-25T19:46:34.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 52332782 0 0) 0x7f62e8003520 con 0x7f630013f710 2024-03-25T19:46:34.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 >> v1:172.21.15.110:6790/0 conn(0x7f63000a98f0 legacy=0x7f63000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 >> v1:172.21.15.178:6789/0 conn(0x7f63001430c0 legacy=0x7f630014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:34.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 1 monclient: found mon.noname-a 2024-03-25T19:46:34.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:34.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:46:34.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63000b44c0 con 0x7f630013f710 2024-03-25T19:46:34.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:34.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:34.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f62feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:34.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f63069e6640 5 monclient: authenticate success, global_id 4494 2024-03-25T19:46:34.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.329+0000 7f63069e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:34.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f62f4001650 con 0x7f630013f710 2024-03-25T19:46:34.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:34.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:34.345 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 10 monclient: dump: 2024-03-25T19:46:34.345 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:34.345 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62f40018a0 con 0x7f630013f710 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f63069e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f63069e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 1 -- 172.21.15.110:0/914598558 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f62f4001c30 con 0x7f630013f710 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:34.346 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f63069e6640 4 set_mon_vals no callback set 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.345+0000 7f62feffd640 10 monclient: dump: 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:34.347 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:34.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.346+0000 7f63069e6640 10 monclient: shutdown 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.346+0000 7f63069e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.346+0000 7f63069e6640 1 -- 172.21.15.110:0/914598558 >> v1:172.21.15.110:6789/0 conn(0x7f630013f710 legacy=0x7f63001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.346+0000 7f63069e6640 1 -- 172.21.15.110:0/914598558 shutdown_connections 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.346+0000 7f63069e6640 1 -- 172.21.15.110:0/914598558 >> 172.21.15.110:0/914598558 conn(0x7f63000b7ea0 msgr2=0x7f63000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.346+0000 7f63069e6640 1 -- 172.21.15.110:0/914598558 shutdown_connections 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.346+0000 7f63069e6640 1 -- 172.21.15.110:0/914598558 wait complete. 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.347+0000 7f63069e6640 10 monclient: build_initial_monmap 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.347+0000 7f63069e6640 10 monclient: monmap: 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:34.350 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.348+0000 7f63069e6640 1 librados: starting msgr at 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.348+0000 7f63069e6640 1 librados: starting objecter 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.348+0000 7f63069e6640 1 Processor -- start 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.348+0000 7f63069e6640 1 -- start start 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.348+0000 7f63069e6640 1 librados: setting wanted keys 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.348+0000 7f63069e6640 1 librados: calling monclient init 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.348+0000 7f63069e6640 10 monclient: init 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:34.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 10 monclient(hunting): picked mon.noname-a con 0x7f63000a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:34.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 10 monclient(hunting): picked mon.noname-b con 0x7f630013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:34.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 10 monclient(hunting): picked mon.noname-c con 0x7f63001430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:34.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63001dad20 con 0x7f63000a98f0 2024-03-25T19:46:34.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63001da9b0 con 0x7f630013f710 2024-03-25T19:46:34.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6300137f90 con 0x7f63001430c0 2024-03-25T19:46:34.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:34.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.349+0000 7f63069e6640 10 monclient(hunting): authenticate will time out at 4565.490531s 2024-03-25T19:46:34.361 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.362+0000 7f62fffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f63000a98f0 0x7f63001db240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34750/0 (socket says 172.21.15.110:34750) 2024-03-25T19:46:34.361 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.362+0000 7f62fffff640 1 -- 172.21.15.110:0/1637720332 learned_addr learned my addr 172.21.15.110:0/1637720332 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:34.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.364+0000 7f62fcff9640 10 client.?.objecter ms_handle_connect 0x7f63000a98f0 2024-03-25T19:46:34.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.364+0000 7f62fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:34.365 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.366+0000 7f62fcff9640 10 client.?.objecter ms_handle_connect 0x7f63001430c0 2024-03-25T19:46:34.365 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.366+0000 7f62fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:34.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.373+0000 7f62fcff9640 10 client.?.objecter ms_handle_connect 0x7f630013f710 2024-03-25T19:46:34.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.373+0000 7f62fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:34.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.376+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541921682 0 0) 0x7f63001dad20 con 0x7f63000a98f0 2024-03-25T19:46:34.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.376+0000 7f62fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.376+0000 7f62fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.376+0000 7f62fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.376+0000 7f62fcff9640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:46:34.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.376+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ec004eb0 con 0x7f63000a98f0 2024-03-25T19:46:34.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.379+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012828196 0 0) 0x7f6300137f90 con 0x7f63001430c0 2024-03-25T19:46:34.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.380+0000 7f62fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.380+0000 7f62fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.380+0000 7f62fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.380+0000 7f62fcff9640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:46:34.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.380+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63001dad20 con 0x7f63001430c0 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.384+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712938564 0 0) 0x7f63001da9b0 con 0x7f630013f710 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.384+0000 7f62fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.384+0000 7f62fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.384+0000 7f62fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.384+0000 7f62fcff9640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:46:34.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.384+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ec005360 con 0x7f630013f710 2024-03-25T19:46:34.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.387+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 427651086 0 0) 0x7f62ec004eb0 con 0x7f63000a98f0 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.387+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63001da9b0 con 0x7f63000a98f0 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.387+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f62f0001970 con 0x7f63000a98f0 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.387+0000 7f62fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.387+0000 7f62fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.387+0000 7f62fcff9640 10 monclient(hunting): dump: 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:34.388 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:34.389 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:34.389 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.387+0000 7f62fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.399+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2111004817 0 0) 0x7f63001da9b0 con 0x7f63000a98f0 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.399+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 >> v1:172.21.15.110:6790/0 conn(0x7f63001430c0 legacy=0x7f6300133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.399+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 >> v1:172.21.15.178:6789/0 conn(0x7f630013f710 legacy=0x7f6300138750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.399+0000 7f62fcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.399+0000 7f62fcff9640 1 monclient: found mon.a 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.399+0000 7f62fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f62fcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6300138120 con 0x7f63000a98f0 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f62fcff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f62fcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f62fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:34.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:46:34.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 10 monclient: _renew_subs 2024-03-25T19:46:34.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:34.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6300138e60 con 0x7f63000a98f0 2024-03-25T19:46:34.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:34.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 10 monclient: _renew_subs 2024-03-25T19:46:34.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:34.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6300134870 con 0x7f63000a98f0 2024-03-25T19:46:34.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.400+0000 7f63069e6640 1 librados: init done 2024-03-25T19:46:34.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.401+0000 7f63069e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:34.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.401+0000 7f63069e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:34.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.401+0000 7f63069e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:34.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.401+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63001dab40 con 0x7f63000a98f0 2024-03-25T19:46:34.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.697+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62f00047e0 con 0x7f63000a98f0 2024-03-25T19:46:34.728 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.697+0000 7f62fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.697+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f62f0004af0 con 0x7f63000a98f0 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.697+0000 7f62fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.697+0000 7f62fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.697+0000 7f62fcff9640 10 monclient: dump: 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:34.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.697+0000 7f62fe7fc640 4 set_mon_vals no callback set 2024-03-25T19:46:34.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.697+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f62f001ab00 con 0x7f63000a98f0 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.698+0000 7f62ff7fe640 10 monclient: get_auth_request con 0x7f62ec06dfa0 auth_method 0 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.698+0000 7f62fcff9640 10 client.4497.objecter ms_handle_connect 0x7f62ec06dfa0 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.703+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f62f0054870 con 0x7f63000a98f0 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.703+0000 7f62fcff9640 10 client.4497.objecter ms_dispatch 0x7f6300000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.703+0000 7f62fcff9640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.703+0000 7f62fcff9640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.703+0000 7f62fcff9640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.703+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62f00501b0 con 0x7f63000a98f0 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.703+0000 7f62fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:34.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.703+0000 7f62fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:34.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.918+0000 7f63069e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.918+0000 7f63069e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.918+0000 7f63069e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:34.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.918+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63000627b0 con 0x7f63000a98f0 2024-03-25T19:46:34.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.954+0000 7f62fcff9640 1 -- 172.21.15.110:0/1637720332 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f62f00522b0 con 0x7f63000a98f0 2024-03-25T19:46:34.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.954+0000 7f62fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.954+0000 7f62fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:34.955 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:46:34.955 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:34.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 10 librados: watch_flush enter 2024-03-25T19:46:34.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 10 librados: watch_flush exit 2024-03-25T19:46:34.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 10 client.4497.objecter successfully canceled tick 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 >> v1:172.21.15.178:6800/1132739058 conn(0x7f62ec06dfa0 legacy=0x7f62ec070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 10 monclient: shutdown 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 >> v1:172.21.15.110:6789/0 conn(0x7f63000a98f0 legacy=0x7f63001db240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 shutdown_connections 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 >> 172.21.15.110:0/1637720332 conn(0x7f63000b7ea0 msgr2=0x7f6300074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 shutdown_connections 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 1 -- 172.21.15.110:0/1637720332 wait complete. 2024-03-25T19:46:34.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:34.957+0000 7f63069e6640 1 librados: shutdown 2024-03-25T19:46:34.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:41.775 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:41.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.889+0000 7f471623e640 10 monclient: get_monmap_and_config 2024-03-25T19:46:41.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.889+0000 7f471623e640 10 monclient: build_initial_monmap 2024-03-25T19:46:41.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.889+0000 7f471623e640 10 monclient: monmap: 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:46:41.890740+0000 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:46:41.890740+0000 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:46:41.890 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.890+0000 7f471623e640 1 Processor -- start 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.890+0000 7f471623e640 1 -- start start 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.890+0000 7f471623e640 10 monclient: init 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 10 monclient(hunting): picked mon.noname-b con 0x7f47100a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 10 monclient(hunting): picked mon.noname-a con 0x7f471013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 10 monclient(hunting): picked mon.noname-c con 0x7f47101430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4710142520 con 0x7f471013f710 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47100b28a0 con 0x7f47101430c0 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47100b2a30 con 0x7f47100a98f0 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:41.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.891+0000 7f471623e640 10 monclient(hunting): authenticate will time out at 4573.032640s 2024-03-25T19:46:41.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.900+0000 7f470f7fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f47100a98f0 0x7f47100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:59678/0 (socket says 172.21.15.110:59678) 2024-03-25T19:46:41.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.900+0000 7f470f7fe640 1 -- 172.21.15.110:0/792611584 learned_addr learned my addr 172.21.15.110:0/792611584 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:41.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.922+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051257321 0 0) 0x7f47100b28a0 con 0x7f47101430c0 2024-03-25T19:46:41.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.922+0000 7f470e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.922+0000 7f470e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.922+0000 7f470e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.922+0000 7f470e7fc640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:46:41.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.922+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f8003520 con 0x7f47101430c0 2024-03-25T19:46:41.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.924+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688094545 0 0) 0x7f4710142520 con 0x7f471013f710 2024-03-25T19:46:41.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.924+0000 7f470e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.924+0000 7f470e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.924+0000 7f470e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.924+0000 7f470e7fc640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:46:41.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.924+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47100b28a0 con 0x7f471013f710 2024-03-25T19:46:41.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.933+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966915162 0 0) 0x7f47100b2a30 con 0x7f47100a98f0 2024-03-25T19:46:41.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.933+0000 7f470e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.933+0000 7f470e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.933+0000 7f470e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.933+0000 7f470e7fc640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:46:41.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.933+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4710142520 con 0x7f47100a98f0 2024-03-25T19:46:41.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1523106221 0 0) 0x7f4710142520 con 0x7f47100a98f0 2024-03-25T19:46:41.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 >> v1:172.21.15.178:6789/0 conn(0x7f47101430c0 legacy=0x7f471014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 >> v1:172.21.15.110:6789/0 conn(0x7f471013f710 legacy=0x7f47101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100b44c0 con 0x7f47100a98f0 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f47000015b0 con 0x7f47100a98f0 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f471623e640 5 monclient: authenticate success, global_id 4508 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:41.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f470e7fc640 10 monclient: dump: 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:41.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.952+0000 7f471623e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:41.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.969+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4700001710 con 0x7f47100a98f0 2024-03-25T19:46:41.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f470e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:41.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f470e7fc640 1 -- 172.21.15.110:0/792611584 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4700001b60 con 0x7f47100a98f0 2024-03-25T19:46:41.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 4 set_mon_vals no callback set 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f470e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f470e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f470e7fc640 10 monclient: dump: 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:41.970 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 10 monclient: shutdown 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 1 -- 172.21.15.110:0/792611584 >> v1:172.21.15.110:6790/0 conn(0x7f47100a98f0 legacy=0x7f47100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 1 -- 172.21.15.110:0/792611584 shutdown_connections 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 1 -- 172.21.15.110:0/792611584 >> 172.21.15.110:0/792611584 conn(0x7f47100b7ea0 msgr2=0x7f47100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 1 -- 172.21.15.110:0/792611584 shutdown_connections 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.970+0000 7f471623e640 1 -- 172.21.15.110:0/792611584 wait complete. 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.971+0000 7f471623e640 10 monclient: build_initial_monmap 2024-03-25T19:46:41.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.971+0000 7f471623e640 10 monclient: monmap: 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.972+0000 7f471623e640 1 librados: starting msgr at 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.972+0000 7f471623e640 1 librados: starting objecter 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.972+0000 7f471623e640 1 Processor -- start 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.972+0000 7f471623e640 1 -- start start 2024-03-25T19:46:41.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.972+0000 7f471623e640 1 librados: setting wanted keys 2024-03-25T19:46:41.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.972+0000 7f471623e640 1 librados: calling monclient init 2024-03-25T19:46:41.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.972+0000 7f471623e640 10 monclient: init 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 10 monclient(hunting): picked mon.noname-b con 0x7f47100a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 10 monclient(hunting): picked mon.noname-a con 0x7f471013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 10 monclient(hunting): picked mon.noname-c con 0x7f47101430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47100abde0 con 0x7f471013f710 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47101d78a0 con 0x7f47100a98f0 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47101d51f0 con 0x7f47101430c0 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:41.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.973+0000 7f471623e640 10 monclient(hunting): authenticate will time out at 4573.114245s 2024-03-25T19:46:41.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.982+0000 7f470f7fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f47100a98f0 0x7f47101d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:39340/0 (socket says 172.21.15.110:39340) 2024-03-25T19:46:41.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.982+0000 7f470f7fe640 1 -- 172.21.15.110:0/3849632874 learned_addr learned my addr 172.21.15.110:0/3849632874 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:41.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.984+0000 7f46f7fff640 10 client.?.objecter ms_handle_connect 0x7f471013f710 2024-03-25T19:46:41.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.984+0000 7f46f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:41.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.985+0000 7f46f7fff640 10 client.?.objecter ms_handle_connect 0x7f47100a98f0 2024-03-25T19:46:41.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.985+0000 7f46f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805962050 0 0) 0x7f47100abde0 con 0x7f471013f710 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46fc004eb0 con 0x7f471013f710 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 client.?.objecter ms_handle_connect 0x7f47101430c0 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584166866 0 0) 0x7f47101d78a0 con 0x7f47100a98f0 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:46:41.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:41.993+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47100abde0 con 0x7f47100a98f0 2024-03-25T19:46:42.008 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.008+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54654772 0 0) 0x7f47100abde0 con 0x7f47100a98f0 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.008+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47101d78a0 con 0x7f47100a98f0 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.009+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4700001ae0 con 0x7f47100a98f0 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.009+0000 7f46f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.009+0000 7f46f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.009+0000 7f46f7fff640 10 monclient(hunting): dump: 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:42.009 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:42.010 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:42.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.009+0000 7f46f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:42.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.012+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664325840 0 0) 0x7f47101d51f0 con 0x7f47101430c0 2024-03-25T19:46:42.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.012+0000 7f46f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.012+0000 7f46f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.012+0000 7f46f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.012+0000 7f46f7fff640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:46:42.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.012+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47100abde0 con 0x7f47101430c0 2024-03-25T19:46:42.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.013+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3604726973 0 0) 0x7f46fc004eb0 con 0x7f471013f710 2024-03-25T19:46:42.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.013+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47101d51f0 con 0x7f471013f710 2024-03-25T19:46:42.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.013+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4704001930 con 0x7f471013f710 2024-03-25T19:46:42.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.013+0000 7f46f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.013+0000 7f46f7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:42.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.013+0000 7f46f7fff640 10 monclient(hunting): dump: 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:42.014 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650052016 0 0) 0x7f47101d78a0 con 0x7f47100a98f0 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 >> v1:172.21.15.110:6790/0 conn(0x7f47101430c0 legacy=0x7f47101e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 >> v1:172.21.15.110:6789/0 conn(0x7f471013f710 legacy=0x7f4710138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 1 monclient: found mon.b 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47101d8440 con 0x7f47100a98f0 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 10 monclient: _finish_auth 0 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f46f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.026+0000 7f471623e640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.027+0000 7f471623e640 10 monclient: _renew_subs 2024-03-25T19:46:42.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.027+0000 7f471623e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.027+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4710138f10 con 0x7f47100a98f0 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.027+0000 7f471623e640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.027+0000 7f471623e640 10 monclient: _renew_subs 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.027+0000 7f471623e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.027+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4710138220 con 0x7f47100a98f0 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.027+0000 7f471623e640 1 librados: init done 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.028+0000 7f471623e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.028+0000 7f471623e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:42.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.028+0000 7f471623e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:42.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.028+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47101d7690 con 0x7f47100a98f0 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.299+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4700004a90 con 0x7f47100a98f0 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.300+0000 7f46f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.300+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4700004da0 con 0x7f47100a98f0 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.300+0000 7f46f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.300+0000 7f46f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.300+0000 7f46f7fff640 10 monclient: dump: 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.300 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:42.301 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:42.301 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:42.301 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:42.301 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:42.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.300+0000 7f470dffb640 4 set_mon_vals no callback set 2024-03-25T19:46:42.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.300+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f470001ad70 con 0x7f47100a98f0 2024-03-25T19:46:42.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.301+0000 7f470effd640 10 monclient: get_auth_request con 0x7f46fc06dfb0 auth_method 0 2024-03-25T19:46:42.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.301+0000 7f46f7fff640 10 client.4513.objecter ms_handle_connect 0x7f46fc06dfb0 2024-03-25T19:46:42.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.305+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f47000548e0 con 0x7f47100a98f0 2024-03-25T19:46:42.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.305+0000 7f46f7fff640 10 client.4513.objecter ms_dispatch 0x7f4710000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:42.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.305+0000 7f46f7fff640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:42.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.305+0000 7f46f7fff640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:42.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.306+0000 7f46f7fff640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T19:46:42.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.308+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47000503f0 con 0x7f47100a98f0 2024-03-25T19:46:42.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.308+0000 7f46f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:42.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.308+0000 7f46f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:42.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.560+0000 7f471623e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:42.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.561+0000 7f471623e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:42.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.561+0000 7f471623e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:46:42.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.561+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47100627b0 con 0x7f47100a98f0 2024-03-25T19:46:42.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.582+0000 7f46f7fff640 1 -- 172.21.15.110:0/3849632874 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f4700052320 con 0x7f47100a98f0 2024-03-25T19:46:42.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.582+0000 7f46f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:42.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.582+0000 7f46f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:42.583 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:46:42.583 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:42.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 10 librados: watch_flush enter 2024-03-25T19:46:42.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 10 librados: watch_flush exit 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 10 client.4513.objecter successfully canceled tick 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 >> v1:172.21.15.178:6800/1132739058 conn(0x7f46fc06dfb0 legacy=0x7f46fc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 10 monclient: shutdown 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 >> v1:172.21.15.178:6789/0 conn(0x7f47100a98f0 legacy=0x7f47101d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 shutdown_connections 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.585+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 >> 172.21.15.110:0/3849632874 conn(0x7f47100b7ea0 msgr2=0x7f4710074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.586+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 shutdown_connections 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.586+0000 7f471623e640 1 -- 172.21.15.110:0/3849632874 wait complete. 2024-03-25T19:46:42.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:42.586+0000 7f471623e640 1 librados: shutdown 2024-03-25T19:46:42.598 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:43.169 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:50.452 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:43.179 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:50.463 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:43.211 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:50.495 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:43.241 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:50.495 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:43.245 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:50.499 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:43.255 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:50.539 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:43.263 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:50.547 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:43.274 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:06:50.528 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:43.291 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:50.575 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:49.404 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:49.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.518+0000 7f3a866f1640 10 monclient: get_monmap_and_config 2024-03-25T19:46:49.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.518+0000 7f3a866f1640 10 monclient: build_initial_monmap 2024-03-25T19:46:49.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.518+0000 7f3a866f1640 10 monclient: monmap: 2024-03-25T19:46:49.518 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:46:49.519596+0000 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:46:49.519596+0000 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:46:49.519 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.519+0000 7f3a866f1640 1 Processor -- start 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.519+0000 7f3a866f1640 1 -- start start 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.519+0000 7f3a866f1640 10 monclient: init 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3a800a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3a8013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3a801430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a80142520 con 0x7f3a801430c0 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a800b28a0 con 0x7f3a8013f710 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a800b2a30 con 0x7f3a800a98f0 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.520+0000 7f3a866f1640 10 monclient(hunting): authenticate will time out at 4580.661306s 2024-03-25T19:46:49.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.525+0000 7f3a7ffff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f3a800a98f0 0x7f3a800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:36290/0 (socket says 172.21.15.110:36290) 2024-03-25T19:46:49.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.525+0000 7f3a7ffff640 1 -- 172.21.15.110:0/2038896996 learned_addr learned my addr 172.21.15.110:0/2038896996 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:49.538 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.539+0000 7f3a7effd640 1 -- 172.21.15.110:0/2038896996 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069608686 0 0) 0x7f3a800b2a30 con 0x7f3a800a98f0 2024-03-25T19:46:49.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.539+0000 7f3a7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:49.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.539+0000 7f3a7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:49.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.539+0000 7f3a7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:49.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.539+0000 7f3a7effd640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:46:49.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.539+0000 7f3a7effd640 1 -- 172.21.15.110:0/2038896996 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68003520 con 0x7f3a800a98f0 2024-03-25T19:46:49.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 1 -- 172.21.15.110:0/2038896996 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1673452142 0 0) 0x7f3a68003520 con 0x7f3a800a98f0 2024-03-25T19:46:49.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 1 -- 172.21.15.110:0/2038896996 >> v1:172.21.15.178:6789/0 conn(0x7f3a8013f710 legacy=0x7f3a801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 1 -- 172.21.15.110:0/2038896996 >> v1:172.21.15.110:6789/0 conn(0x7f3a801430c0 legacy=0x7f3a8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:49.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 1 monclient: found mon.noname-b 2024-03-25T19:46:49.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 1 -- 172.21.15.110:0/2038896996 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a800b44c0 con 0x7f3a800a98f0 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 10 monclient: _finish_auth 0 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 1 -- 172.21.15.110:0/2038896996 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3a70001690 con 0x7f3a800a98f0 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a7effd640 10 monclient: dump: 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:49.551 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:49.552 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:49.552 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:49.552 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:49.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a866f1640 5 monclient: authenticate success, global_id 4514 2024-03-25T19:46:49.552 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.550+0000 7f3a866f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:49.570 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.571+0000 7f3a7effd640 1 -- 172.21.15.110:0/2038896996 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a700017f0 con 0x7f3a800a98f0 2024-03-25T19:46:49.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.571+0000 7f3a7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:49.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.571+0000 7f3a866f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:49.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.571+0000 7f3a866f1640 4 set_mon_vals no callback set 2024-03-25T19:46:49.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 10 monclient: shutdown 2024-03-25T19:46:49.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:49.571 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 1 -- 172.21.15.110:0/2038896996 >> v1:172.21.15.110:6790/0 conn(0x7f3a800a98f0 legacy=0x7f3a800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.572 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 1 -- 172.21.15.110:0/2038896996 shutdown_connections 2024-03-25T19:46:49.572 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 1 -- 172.21.15.110:0/2038896996 >> 172.21.15.110:0/2038896996 conn(0x7f3a800b7ea0 msgr2=0x7f3a800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:49.572 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 1 -- 172.21.15.110:0/2038896996 shutdown_connections 2024-03-25T19:46:49.572 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 1 -- 172.21.15.110:0/2038896996 wait complete. 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 10 monclient: build_initial_monmap 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.572+0000 7f3a866f1640 10 monclient: monmap: 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:46:49.573 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:46:49.574 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:49.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.573+0000 7f3a866f1640 1 librados: starting msgr at 2024-03-25T19:46:49.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.573+0000 7f3a866f1640 1 librados: starting objecter 2024-03-25T19:46:49.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.573+0000 7f3a866f1640 1 Processor -- start 2024-03-25T19:46:49.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.573+0000 7f3a866f1640 1 -- start start 2024-03-25T19:46:49.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.573+0000 7f3a866f1640 1 librados: setting wanted keys 2024-03-25T19:46:49.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.573+0000 7f3a866f1640 1 librados: calling monclient init 2024-03-25T19:46:49.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.573+0000 7f3a866f1640 10 monclient: init 2024-03-25T19:46:49.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:49.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:49.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3a800a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:49.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3a8013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:49.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3a801430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:49.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a80136890 con 0x7f3a801430c0 2024-03-25T19:46:49.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a80136d90 con 0x7f3a8013f710 2024-03-25T19:46:49.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a80139840 con 0x7f3a800a98f0 2024-03-25T19:46:49.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:49.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.574+0000 7f3a866f1640 10 monclient(hunting): authenticate will time out at 4580.715375s 2024-03-25T19:46:49.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.577+0000 7f3a84c8e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3a801430c0 0x7f3a801e1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40366/0 (socket says 172.21.15.110:40366) 2024-03-25T19:46:49.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.577+0000 7f3a84c8e640 1 -- 172.21.15.110:0/14153783 learned_addr learned my addr 172.21.15.110:0/14153783 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:49.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.578+0000 7f3a7cff9640 10 client.?.objecter ms_handle_connect 0x7f3a801430c0 2024-03-25T19:46:49.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.578+0000 7f3a7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:49.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.588+0000 7f3a7cff9640 10 client.?.objecter ms_handle_connect 0x7f3a800a98f0 2024-03-25T19:46:49.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.588+0000 7f3a7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:49.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.590+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329979972 0 0) 0x7f3a80136890 con 0x7f3a801430c0 2024-03-25T19:46:49.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.590+0000 7f3a7cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:49.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.590+0000 7f3a7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:49.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.590+0000 7f3a7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:49.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.590+0000 7f3a7cff9640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:46:49.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.590+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6c004eb0 con 0x7f3a801430c0 2024-03-25T19:46:49.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.593+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237586478 0 0) 0x7f3a80139840 con 0x7f3a800a98f0 2024-03-25T19:46:49.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.593+0000 7f3a7cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:49.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.593+0000 7f3a7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:49.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.593+0000 7f3a7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:49.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.593+0000 7f3a7cff9640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:46:49.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.593+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a80136890 con 0x7f3a800a98f0 2024-03-25T19:46:49.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.598+0000 7f3a7cff9640 10 client.?.objecter ms_handle_connect 0x7f3a8013f710 2024-03-25T19:46:49.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.598+0000 7f3a7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.604+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2708262010 0 0) 0x7f3a6c004eb0 con 0x7f3a801430c0 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.604+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a80139840 con 0x7f3a801430c0 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.604+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3a78001c30 con 0x7f3a801430c0 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.604+0000 7f3a7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.604+0000 7f3a7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.604+0000 7f3a7cff9640 10 monclient(hunting): dump: 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:49.604 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:49.605 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.605 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:49.605 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:49.605 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:49.605 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:49.605 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:49.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.604+0000 7f3a7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:49.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 28978124 0 0) 0x7f3a80139840 con 0x7f3a801430c0 2024-03-25T19:46:49.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 >> v1:172.21.15.110:6790/0 conn(0x7f3a800a98f0 legacy=0x7f3a80139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 >> v1:172.21.15.178:6789/0 conn(0x7f3a8013f710 legacy=0x7f3a801370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:49.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 1 monclient: found mon.a 2024-03-25T19:46:49.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a801399d0 con 0x7f3a801430c0 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 5 monclient: authenticate success, global_id 4509 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 10 monclient: _renew_subs 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a801377d0 con 0x7f3a801430c0 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 10 monclient: _renew_subs 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a800b6e50 con 0x7f3a801430c0 2024-03-25T19:46:49.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.619+0000 7f3a866f1640 1 librados: init done 2024-03-25T19:46:49.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.620+0000 7f3a866f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:49.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.620+0000 7f3a866f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:49.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.620+0000 7f3a866f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:49.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.620+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a800627b0 con 0x7f3a801430c0 2024-03-25T19:46:49.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.934+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a78003970 con 0x7f3a801430c0 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.934+0000 7f3a7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.934+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3a78003d00 con 0x7f3a801430c0 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.934+0000 7f3a7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.934+0000 7f3a7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.934+0000 7f3a7cff9640 10 monclient: dump: 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:49.935 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:49.936 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:49.936 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:49.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.934+0000 7f3a7e7fc640 4 set_mon_vals no callback set 2024-03-25T19:46:49.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.935+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f3a78003f20 con 0x7f3a801430c0 2024-03-25T19:46:49.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.936+0000 7f3a7ffff640 10 monclient: get_auth_request con 0x7f3a6c06dd10 auth_method 0 2024-03-25T19:46:49.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.936+0000 7f3a7cff9640 10 client.4509.objecter ms_handle_connect 0x7f3a6c06dd10 2024-03-25T19:46:49.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.941+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f3a78054a30 con 0x7f3a801430c0 2024-03-25T19:46:49.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.941+0000 7f3a7cff9640 10 client.4509.objecter ms_dispatch 0x7f3a80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:49.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.941+0000 7f3a7cff9640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:49.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.941+0000 7f3a7cff9640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:49.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.941+0000 7f3a7cff9640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T19:46:49.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.941+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a78050540 con 0x7f3a801430c0 2024-03-25T19:46:49.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.941+0000 7f3a7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:49.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:49.941+0000 7f3a7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.148+0000 7f3a866f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.148+0000 7f3a866f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.148+0000 7f3a866f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:46:50.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.148+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a800b7150 con 0x7f3a801430c0 2024-03-25T19:46:50.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.218+0000 7f3a7cff9640 1 -- 172.21.15.110:0/14153783 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f3a78052470 con 0x7f3a801430c0 2024-03-25T19:46:50.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.218+0000 7f3a7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:50.218 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.218+0000 7f3a7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:50.219 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:46:50.219 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:50.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 10 librados: watch_flush enter 2024-03-25T19:46:50.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 10 librados: watch_flush exit 2024-03-25T19:46:50.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T19:46:50.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 10 client.4509.objecter successfully canceled tick 2024-03-25T19:46:50.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 >> v1:172.21.15.178:6800/1132739058 conn(0x7f3a6c06dd10 legacy=0x7f3a6c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 10 monclient: shutdown 2024-03-25T19:46:50.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:50.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 >> v1:172.21.15.110:6789/0 conn(0x7f3a801430c0 legacy=0x7f3a801e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 shutdown_connections 2024-03-25T19:46:50.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 >> 172.21.15.110:0/14153783 conn(0x7f3a800b7ea0 msgr2=0x7f3a8013d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:50.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 shutdown_connections 2024-03-25T19:46:50.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 1 -- 172.21.15.110:0/14153783 wait complete. 2024-03-25T19:46:50.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:50.220+0000 7f3a866f1640 1 librados: shutdown 2024-03-25T19:46:50.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:57.038 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:57.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.152+0000 7f58ce6d4640 10 monclient: get_monmap_and_config 2024-03-25T19:46:57.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.152+0000 7f58ce6d4640 10 monclient: build_initial_monmap 2024-03-25T19:46:57.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.152+0000 7f58ce6d4640 10 monclient: monmap: 2024-03-25T19:46:57.152 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:57.152 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:57.153 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:46:57.153456+0000 2024-03-25T19:46:57.153 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:46:57.153456+0000 2024-03-25T19:46:57.153 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:57.153 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:57.153 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:57.153 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:46:57.153 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:46:57.153 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:57.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.153+0000 7f58ce6d4640 1 Processor -- start 2024-03-25T19:46:57.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.153+0000 7f58ce6d4640 1 -- start start 2024-03-25T19:46:57.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.153+0000 7f58ce6d4640 10 monclient: init 2024-03-25T19:46:57.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:57.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:57.154 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f58c80a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:57.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f58c813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:57.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f58c81430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:57.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c8142520 con 0x7f58c81430c0 2024-03-25T19:46:57.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c80b28a0 con 0x7f58c80a98f0 2024-03-25T19:46:57.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c80b2a30 con 0x7f58c813f710 2024-03-25T19:46:57.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:57.155 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.154+0000 7f58ce6d4640 10 monclient(hunting): authenticate will time out at 4588.295265s 2024-03-25T19:46:57.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.156+0000 7f58c77fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f58c813f710 0x7f58c81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:47290/0 (socket says 172.21.15.110:47290) 2024-03-25T19:46:57.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.156+0000 7f58c77fe640 1 -- 172.21.15.110:0/1004355311 learned_addr learned my addr 172.21.15.110:0/1004355311 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:57.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.165+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894933598 0 0) 0x7f58c80b2a30 con 0x7f58c813f710 2024-03-25T19:46:57.165 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.165+0000 7f58c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:57.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.165+0000 7f58c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:57.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.165+0000 7f58c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:57.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.165+0000 7f58c6ffd640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:46:57.166 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.165+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b0003520 con 0x7f58c813f710 2024-03-25T19:46:57.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.171+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080902931 0 0) 0x7f58c8142520 con 0x7f58c81430c0 2024-03-25T19:46:57.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.171+0000 7f58c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:57.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.171+0000 7f58c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:57.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.171+0000 7f58c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:57.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.171+0000 7f58c6ffd640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:46:57.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.171+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c80b2a30 con 0x7f58c81430c0 2024-03-25T19:46:57.191 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3952656246 0 0) 0x7f58c80b2a30 con 0x7f58c81430c0 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 >> v1:172.21.15.110:6790/0 conn(0x7f58c813f710 legacy=0x7f58c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 >> v1:172.21.15.178:6789/0 conn(0x7f58c80a98f0 legacy=0x7f58c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c80b44c0 con 0x7f58c81430c0 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58c00019f0 con 0x7f58c81430c0 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58ce6d4640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58ce6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:57.192 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58c6ffd640 10 monclient: dump: 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:57.193 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.191+0000 7f58ce6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.202+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58c0001b50 con 0x7f58c81430c0 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.202+0000 7f58c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.202+0000 7f58c6ffd640 1 -- 172.21.15.110:0/1004355311 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58c0001f20 con 0x7f58c81430c0 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.202+0000 7f58c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.202+0000 7f58c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.202+0000 7f58c6ffd640 10 monclient: dump: 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.202 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:57.203 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:57.203 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:57.203 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:57.203 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:57.203 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:57.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.202+0000 7f58ce6d4640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:57.203 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.202+0000 7f58ce6d4640 4 set_mon_vals no callback set 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 10 monclient: shutdown 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 1 -- 172.21.15.110:0/1004355311 >> v1:172.21.15.110:6789/0 conn(0x7f58c81430c0 legacy=0x7f58c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 1 -- 172.21.15.110:0/1004355311 shutdown_connections 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 1 -- 172.21.15.110:0/1004355311 >> 172.21.15.110:0/1004355311 conn(0x7f58c80b7ea0 msgr2=0x7f58c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 1 -- 172.21.15.110:0/1004355311 shutdown_connections 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 1 -- 172.21.15.110:0/1004355311 wait complete. 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 10 monclient: build_initial_monmap 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.203+0000 7f58ce6d4640 10 monclient: monmap: 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:46:57.205 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.204+0000 7f58ce6d4640 1 librados: starting msgr at 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.204+0000 7f58ce6d4640 1 librados: starting objecter 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.204+0000 7f58ce6d4640 1 Processor -- start 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.204+0000 7f58ce6d4640 1 -- start start 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.204+0000 7f58ce6d4640 1 librados: setting wanted keys 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.204+0000 7f58ce6d4640 1 librados: calling monclient init 2024-03-25T19:46:57.206 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.204+0000 7f58ce6d4640 10 monclient: init 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f58c80a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f58c813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f58c81430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c8139910 con 0x7f58c81430c0 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c80ab8b0 con 0x7f58c80a98f0 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c81db470 con 0x7f58c813f710 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:57.207 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.205+0000 7f58ce6d4640 10 monclient(hunting): authenticate will time out at 4588.346063s 2024-03-25T19:46:57.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.216+0000 7f58c77fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f58c813f710 0x7f58c80b0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:47340/0 (socket says 172.21.15.110:47340) 2024-03-25T19:46:57.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.216+0000 7f58c77fe640 1 -- 172.21.15.110:0/213957742 learned_addr learned my addr 172.21.15.110:0/213957742 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:46:57.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.217+0000 7f58c4ff9640 10 client.?.objecter ms_handle_connect 0x7f58c813f710 2024-03-25T19:46:57.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.217+0000 7f58c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:57.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.219+0000 7f58c4ff9640 10 client.?.objecter ms_handle_connect 0x7f58c80a98f0 2024-03-25T19:46:57.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.219+0000 7f58c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:57.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.220+0000 7f58c4ff9640 10 client.?.objecter ms_handle_connect 0x7f58c81430c0 2024-03-25T19:46:57.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.220+0000 7f58c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:57.224 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.224+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121360194 0 0) 0x7f58c81db470 con 0x7f58c813f710 2024-03-25T19:46:57.224 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.224+0000 7f58c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:57.224 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.224+0000 7f58c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:57.224 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.224+0000 7f58c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:57.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.224+0000 7f58c4ff9640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:46:57.225 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.224+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b4004eb0 con 0x7f58c813f710 2024-03-25T19:46:57.228 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.228+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483987794 0 0) 0x7f58c8139910 con 0x7f58c81430c0 2024-03-25T19:46:57.228 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.228+0000 7f58c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:57.228 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.228+0000 7f58c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:57.228 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.228+0000 7f58c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:57.228 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.228+0000 7f58c4ff9640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:46:57.228 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.228+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c81db470 con 0x7f58c81430c0 2024-03-25T19:46:57.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.247+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719254160 0 0) 0x7f58c80ab8b0 con 0x7f58c80a98f0 2024-03-25T19:46:57.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.247+0000 7f58c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:57.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.247+0000 7f58c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:57.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.247+0000 7f58c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:57.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.247+0000 7f58c4ff9640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:46:57.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.247+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c8139910 con 0x7f58c80a98f0 2024-03-25T19:46:57.258 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.258+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1491404527 0 0) 0x7f58b4004eb0 con 0x7f58c813f710 2024-03-25T19:46:57.258 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.258+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58c80ab8b0 con 0x7f58c813f710 2024-03-25T19:46:57.258 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.258+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58bc001950 con 0x7f58c813f710 2024-03-25T19:46:57.258 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.258+0000 7f58c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:57.258 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.258+0000 7f58c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:57.258 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.258+0000 7f58c4ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:57.258 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:57.258 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:57.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.258+0000 7f58c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.264+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3211098592 0 0) 0x7f58c81db470 con 0x7f58c81430c0 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.264+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58b4004eb0 con 0x7f58c81430c0 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.264+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58c0001e60 con 0x7f58c81430c0 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.264+0000 7f58c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.264+0000 7f58c4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.264+0000 7f58c4ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.264 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.265 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:57.265 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:57.265 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:57.265 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:57.265 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:57.265 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2465916808 0 0) 0x7f58c80ab8b0 con 0x7f58c813f710 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 >> v1:172.21.15.178:6789/0 conn(0x7f58c80a98f0 legacy=0x7f58c81daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 >> v1:172.21.15.110:6789/0 conn(0x7f58c81430c0 legacy=0x7f58c8137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 1 monclient: found mon.c 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c81db110 con 0x7f58c813f710 2024-03-25T19:46:57.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 5 monclient: authenticate success, global_id 4523 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 10 monclient: _renew_subs 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c80b0d50 con 0x7f58c813f710 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 10 monclient: _renew_subs 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c80b0930 con 0x7f58c813f710 2024-03-25T19:46:57.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.270+0000 7f58ce6d4640 1 librados: init done 2024-03-25T19:46:57.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.271+0000 7f58ce6d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:57.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.271+0000 7f58ce6d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:57.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.271+0000 7f58ce6d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:57.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.271+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c80b10f0 con 0x7f58c813f710 2024-03-25T19:46:57.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.605+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58bc0047e0 con 0x7f58c813f710 2024-03-25T19:46:57.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.605+0000 7f58c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.605+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58bc004af0 con 0x7f58c813f710 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.605+0000 7f58c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.605+0000 7f58c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.605+0000 7f58c4ff9640 10 monclient: dump: 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:46:57.606 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:46:57.607 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:46:57.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.605+0000 7f58c67fc640 4 set_mon_vals no callback set 2024-03-25T19:46:57.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.606+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f58bc01aac0 con 0x7f58c813f710 2024-03-25T19:46:57.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.607+0000 7f58c7fff640 10 monclient: get_auth_request con 0x7f58b406df20 auth_method 0 2024-03-25T19:46:57.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.607+0000 7f58c4ff9640 10 client.4523.objecter ms_handle_connect 0x7f58b406df20 2024-03-25T19:46:57.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.612+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f58bc054800 con 0x7f58c813f710 2024-03-25T19:46:57.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.612+0000 7f58c4ff9640 10 client.4523.objecter ms_dispatch 0x7f58c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:57.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.612+0000 7f58c4ff9640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:57.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.612+0000 7f58c4ff9640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:57.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.612+0000 7f58c4ff9640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T19:46:57.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.612+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58bc050140 con 0x7f58c813f710 2024-03-25T19:46:57.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.612+0000 7f58c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:57.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.612+0000 7f58c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:57.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.835+0000 7f58ce6d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.835+0000 7f58ce6d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.835+0000 7f58ce6d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:46:57.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.835+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58c81db600 con 0x7f58c813f710 2024-03-25T19:46:57.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.884+0000 7f58c4ff9640 1 -- 172.21.15.110:0/213957742 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f58bc052240 con 0x7f58c813f710 2024-03-25T19:46:57.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.884+0000 7f58c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.884+0000 7f58c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:57.884 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:46:57.884 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 10 librados: watch_flush enter 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 10 librados: watch_flush exit 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 10 client.4523.objecter successfully canceled tick 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 >> v1:172.21.15.178:6800/1132739058 conn(0x7f58b406df20 legacy=0x7f58b40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 10 monclient: shutdown 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 >> v1:172.21.15.110:6790/0 conn(0x7f58c813f710 legacy=0x7f58c80b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 shutdown_connections 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 >> 172.21.15.110:0/213957742 conn(0x7f58c80b7ea0 msgr2=0x7f58c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.886+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 shutdown_connections 2024-03-25T19:46:57.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.887+0000 7f58ce6d4640 1 -- 172.21.15.110:0/213957742 wait complete. 2024-03-25T19:46:57.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:46:57.887+0000 7f58ce6d4640 1 librados: shutdown 2024-03-25T19:46:57.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:03.171 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.455 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:03.220 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.504 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:03.265 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.549 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:03.285 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.569 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:03.295 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.579 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:03.297 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:10.552 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:03.309 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:10.564 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:03.321 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.605 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:03.333 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:10.588 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:04.706 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:04.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.820+0000 7f5d0d64a640 10 monclient: get_monmap_and_config 2024-03-25T19:47:04.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.820+0000 7f5d0d64a640 10 monclient: build_initial_monmap 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.821+0000 7f5d0d64a640 10 monclient: monmap: 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:47:04.821932+0000 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:47:04.821932+0000 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:47:04.821 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.821+0000 7f5d0d64a640 1 Processor -- start 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.821+0000 7f5d0d64a640 1 -- start start 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.821+0000 7f5d0d64a640 10 monclient: init 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 10 monclient(hunting): picked mon.noname-b con 0x7f5d080a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 10 monclient(hunting): picked mon.noname-c con 0x7f5d0813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 10 monclient(hunting): picked mon.noname-a con 0x7f5d081430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d08142520 con 0x7f5d081430c0 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080b28a0 con 0x7f5d0813f710 2024-03-25T19:47:04.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080b2a30 con 0x7f5d080a98f0 2024-03-25T19:47:04.824 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:04.824 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.822+0000 7f5d0d64a640 10 monclient(hunting): authenticate will time out at 4595.963808s 2024-03-25T19:47:04.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.835+0000 7f5cfe7fc640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f5d0813f710 0x7f5d081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:42194/0 (socket says 172.21.15.110:42194) 2024-03-25T19:47:04.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.835+0000 7f5cfe7fc640 1 -- 172.21.15.110:0/1172560615 learned_addr learned my addr 172.21.15.110:0/1172560615 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:04.851 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.851+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321975781 0 0) 0x7f5d080b2a30 con 0x7f5d080a98f0 2024-03-25T19:47:04.851 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.851+0000 7f5d067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.851 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.851+0000 7f5d067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.851 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.851+0000 7f5d067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.851 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.851+0000 7f5d067fc640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:47:04.851 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.851+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf4003520 con 0x7f5d080a98f0 2024-03-25T19:47:04.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800089518 0 0) 0x7f5d080b28a0 con 0x7f5d0813f710 2024-03-25T19:47:04.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:47:04.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d080b2a30 con 0x7f5d0813f710 2024-03-25T19:47:04.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912054933 0 0) 0x7f5d08142520 con 0x7f5d081430c0 2024-03-25T19:47:04.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:47:04.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.853+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d080b28a0 con 0x7f5d081430c0 2024-03-25T19:47:04.883 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1201640839 0 0) 0x7f5d080b28a0 con 0x7f5d081430c0 2024-03-25T19:47:04.883 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 >> v1:172.21.15.110:6790/0 conn(0x7f5d080a98f0 legacy=0x7f5d080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.883 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 >> v1:172.21.15.178:6789/0 conn(0x7f5d0813f710 legacy=0x7f5d081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.883 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:04.883 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 1 monclient: found mon.noname-a 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080b44c0 con 0x7f5d081430c0 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5d00001930 con 0x7f5d081430c0 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d067fc640 10 monclient: dump: 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:04.884 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.885 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:04.885 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:04.885 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:04.885 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:04.885 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:04.885 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d0d64a640 5 monclient: authenticate success, global_id 4518 2024-03-25T19:47:04.885 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.883+0000 7f5d0d64a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:04.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.935+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d00001a90 con 0x7f5d081430c0 2024-03-25T19:47:04.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.935+0000 7f5d067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:04.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.935+0000 7f5d067fc640 1 -- 172.21.15.110:0/1172560615 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5d00001ea0 con 0x7f5d081430c0 2024-03-25T19:47:04.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.935+0000 7f5d067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.935+0000 7f5d067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:04.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.935+0000 7f5d067fc640 10 monclient: dump: 2024-03-25T19:47:04.935 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:04.935 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.935+0000 7f5d0d64a640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:04.936 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.935+0000 7f5d0d64a640 4 set_mon_vals no callback set 2024-03-25T19:47:04.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.936+0000 7f5d0d64a640 10 monclient: shutdown 2024-03-25T19:47:04.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.936+0000 7f5d0d64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:04.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.936+0000 7f5d0d64a640 1 -- 172.21.15.110:0/1172560615 >> v1:172.21.15.110:6789/0 conn(0x7f5d081430c0 legacy=0x7f5d0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.936+0000 7f5d0d64a640 1 -- 172.21.15.110:0/1172560615 shutdown_connections 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.936+0000 7f5d0d64a640 1 -- 172.21.15.110:0/1172560615 >> 172.21.15.110:0/1172560615 conn(0x7f5d080b7ea0 msgr2=0x7f5d080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.936+0000 7f5d0d64a640 1 -- 172.21.15.110:0/1172560615 shutdown_connections 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.936+0000 7f5d0d64a640 1 -- 172.21.15.110:0/1172560615 wait complete. 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.937+0000 7f5d0d64a640 10 monclient: build_initial_monmap 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.937+0000 7f5d0d64a640 10 monclient: monmap: 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:47:04.938 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.937+0000 7f5d0d64a640 1 librados: starting msgr at 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.937+0000 7f5d0d64a640 1 librados: starting objecter 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.937+0000 7f5d0d64a640 1 Processor -- start 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 1 -- start start 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 1 librados: setting wanted keys 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 1 librados: calling monclient init 2024-03-25T19:47:04.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 10 monclient: init 2024-03-25T19:47:04.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:04.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:04.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 10 monclient(hunting): picked mon.noname-a con 0x7f5d080a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:04.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 10 monclient(hunting): picked mon.noname-c con 0x7f5d0813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:04.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 10 monclient(hunting): picked mon.noname-b con 0x7f5d081430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:04.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d081349e0 con 0x7f5d080a98f0 2024-03-25T19:47:04.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080ab950 con 0x7f5d081430c0 2024-03-25T19:47:04.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d08138770 con 0x7f5d0813f710 2024-03-25T19:47:04.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:04.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.938+0000 7f5d0d64a640 10 monclient(hunting): authenticate will time out at 4596.079530s 2024-03-25T19:47:04.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.943+0000 7f5d077fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f5d081430c0 0x7f5d081342d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:42254/0 (socket says 172.21.15.110:42254) 2024-03-25T19:47:04.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.943+0000 7f5d077fe640 1 -- 172.21.15.110:0/50391444 learned_addr learned my addr 172.21.15.110:0/50391444 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:04.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.944+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d081430c0 2024-03-25T19:47:04.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.944+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.945+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d0813f710 2024-03-25T19:47:04.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.945+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.954+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408035391 0 0) 0x7f5d080ab950 con 0x7f5d081430c0 2024-03-25T19:47:04.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.954+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.954+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.954+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.954+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:47:04.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.954+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce8004eb0 con 0x7f5d081430c0 2024-03-25T19:47:04.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.957+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d080a98f0 2024-03-25T19:47:04.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.957+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.958+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835828519 0 0) 0x7f5d08138770 con 0x7f5d0813f710 2024-03-25T19:47:04.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.958+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.958+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.958+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.958+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:47:04.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.958+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d080ab950 con 0x7f5d0813f710 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.963+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 599817427 0 0) 0x7f5ce8004eb0 con 0x7f5d081430c0 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.963+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d08138770 con 0x7f5d081430c0 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.963+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5d00001ea0 con 0x7f5d081430c0 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.963+0000 7f5cfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.963+0000 7f5cfffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.963+0000 7f5cfffff640 10 monclient(hunting): dump: 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:04.963 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:04.964 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:04.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.963+0000 7f5cfffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:04.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.973+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215320529 0 0) 0x7f5d081349e0 con 0x7f5d080a98f0 2024-03-25T19:47:04.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.973+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.973+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.973+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.973+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:47:04.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.973+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce8004eb0 con 0x7f5d080a98f0 2024-03-25T19:47:04.983 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.983+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2977894835 0 0) 0x7f5d080ab950 con 0x7f5d0813f710 2024-03-25T19:47:04.983 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.983+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d081349e0 con 0x7f5d0813f710 2024-03-25T19:47:04.983 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.983+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5cf0001930 con 0x7f5d0813f710 2024-03-25T19:47:04.983 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.983+0000 7f5cfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.983 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.983+0000 7f5cfffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:04.983 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.983+0000 7f5cfffff640 10 monclient(hunting): dump: 2024-03-25T19:47:04.983 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:04.983 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:04.984 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:04.984 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:04.984 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.984 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:04.984 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:04.984 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:04.984 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:04.984 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:04.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 785537229 0 0) 0x7f5d08138770 con 0x7f5d081430c0 2024-03-25T19:47:04.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 >> v1:172.21.15.110:6790/0 conn(0x7f5d0813f710 legacy=0x7f5d08133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 >> v1:172.21.15.110:6789/0 conn(0x7f5d080a98f0 legacy=0x7f5d081da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 1 monclient: found mon.b 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080b1670 con 0x7f5d081430c0 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 10 monclient: _finish_auth 0 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5cfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 5 monclient: authenticate success, global_id 4531 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 10 monclient: _renew_subs 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:04.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d081378e0 con 0x7f5d081430c0 2024-03-25T19:47:04.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:04.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 10 monclient: _renew_subs 2024-03-25T19:47:04.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:04.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d081daf90 con 0x7f5d081430c0 2024-03-25T19:47:04.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.985+0000 7f5d0d64a640 1 librados: init done 2024-03-25T19:47:04.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.986+0000 7f5d0d64a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:04.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.987+0000 7f5d0d64a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:04.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.987+0000 7f5d0d64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:04.988 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:04.987+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d08138900 con 0x7f5d081430c0 2024-03-25T19:47:05.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.262+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d00003c20 con 0x7f5d081430c0 2024-03-25T19:47:05.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.262+0000 7f5cfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:05.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.262+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5d00002100 con 0x7f5d081430c0 2024-03-25T19:47:05.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.262+0000 7f5cfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.262+0000 7f5cfffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.262+0000 7f5cfffff640 10 monclient: dump: 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:05.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.262+0000 7f5d05ffb640 4 set_mon_vals no callback set 2024-03-25T19:47:05.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.263+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f5d00004070 con 0x7f5d081430c0 2024-03-25T19:47:05.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.264+0000 7f5d06ffd640 10 monclient: get_auth_request con 0x7f5ce806dfb0 auth_method 0 2024-03-25T19:47:05.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.264+0000 7f5cfffff640 10 client.4531.objecter ms_handle_connect 0x7f5ce806dfb0 2024-03-25T19:47:05.267 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.267+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f5d00054cd0 con 0x7f5d081430c0 2024-03-25T19:47:05.267 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.267+0000 7f5cfffff640 10 client.4531.objecter ms_dispatch 0x7f5d08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:05.267 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.267+0000 7f5cfffff640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:05.267 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.267+0000 7f5cfffff640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:05.267 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.267+0000 7f5cfffff640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T19:47:05.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.268+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d00050610 con 0x7f5d081430c0 2024-03-25T19:47:05.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.268+0000 7f5cfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:05.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.268+0000 7f5cfffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:05.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.519+0000 7f5d0d64a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:05.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.519+0000 7f5d0d64a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:05.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.519+0000 7f5d0d64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:05.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.519+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d080627b0 con 0x7f5d081430c0 2024-03-25T19:47:05.546 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.546+0000 7f5cfffff640 1 -- 172.21.15.110:0/50391444 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f5d00052710 con 0x7f5d081430c0 2024-03-25T19:47:05.546 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.546+0000 7f5cfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:05.546 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.546+0000 7f5cfffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:05.547 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:47:05.547 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:05.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 10 librados: watch_flush enter 2024-03-25T19:47:05.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 10 librados: watch_flush exit 2024-03-25T19:47:05.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T19:47:05.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 10 client.4531.objecter successfully canceled tick 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 >> v1:172.21.15.178:6800/1132739058 conn(0x7f5ce806dfb0 legacy=0x7f5ce8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 10 monclient: shutdown 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 >> v1:172.21.15.178:6789/0 conn(0x7f5d081430c0 legacy=0x7f5d081342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 shutdown_connections 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 >> 172.21.15.110:0/50391444 conn(0x7f5d080b7ea0 msgr2=0x7f5d08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 shutdown_connections 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 1 -- 172.21.15.110:0/50391444 wait complete. 2024-03-25T19:47:05.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:05.549+0000 7f5d0d64a640 1 librados: shutdown 2024-03-25T19:47:05.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:12.367 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:12.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.484+0000 7f0519625640 10 monclient: get_monmap_and_config 2024-03-25T19:47:12.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.484+0000 7f0519625640 10 monclient: build_initial_monmap 2024-03-25T19:47:12.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.484+0000 7f0519625640 10 monclient: monmap: 2024-03-25T19:47:12.484 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:12.484 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:12.484 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:47:12.485377+0000 2024-03-25T19:47:12.484 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:47:12.485377+0000 2024-03-25T19:47:12.484 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:12.485 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:12.485 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:12.485 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:47:12.485 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:47:12.485 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:12.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.485+0000 7f0519625640 1 Processor -- start 2024-03-25T19:47:12.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.485+0000 7f0519625640 1 -- start start 2024-03-25T19:47:12.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.485+0000 7f0519625640 10 monclient: init 2024-03-25T19:47:12.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.485+0000 7f0519625640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:12.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.485+0000 7f0519625640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.485+0000 7f0519625640 10 monclient(hunting): picked mon.noname-c con 0x7f05140a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.485+0000 7f0519625640 10 monclient(hunting): picked mon.noname-b con 0x7f051413f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.486+0000 7f0519625640 10 monclient(hunting): picked mon.noname-a con 0x7f05141430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.486+0000 7f0519625640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0514142520 con 0x7f05141430c0 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.486+0000 7f0519625640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140b28a0 con 0x7f05140a98f0 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.486+0000 7f0519625640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140b2a30 con 0x7f051413f710 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.486+0000 7f0519625640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.486+0000 7f0519625640 10 monclient(hunting): authenticate will time out at 4603.626967s 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.486+0000 7f0512ffd640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f05140a98f0 0x7f05140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:47974/0 (socket says 172.21.15.110:47974) 2024-03-25T19:47:12.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.486+0000 7f0512ffd640 1 -- 172.21.15.110:0/2539753547 learned_addr learned my addr 172.21.15.110:0/2539753547 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:12.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.496+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600874019 0 0) 0x7f05140b28a0 con 0x7f05140a98f0 2024-03-25T19:47:12.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.496+0000 7f0511ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.496+0000 7f0511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.496+0000 7f0511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.496+0000 7f0511ffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:47:12.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.496+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc003520 con 0x7f05140a98f0 2024-03-25T19:47:12.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1410883863 0 0) 0x7f04fc003520 con 0x7f05140a98f0 2024-03-25T19:47:12.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 >> v1:172.21.15.110:6790/0 conn(0x7f051413f710 legacy=0x7f05141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 >> v1:172.21.15.110:6789/0 conn(0x7f05141430c0 legacy=0x7f051414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:47:12.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 1 monclient: found mon.noname-c 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140b44c0 con 0x7f05140a98f0 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 10 monclient: _finish_auth 0 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0504001690 con 0x7f05140a98f0 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0511ffb640 10 monclient: dump: 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.504 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.505 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:12.505 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:12.505 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:12.505 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:12.505 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:12.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0519625640 5 monclient: authenticate success, global_id 4534 2024-03-25T19:47:12.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.503+0000 7f0519625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:12.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.518+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0504001860 con 0x7f05140a98f0 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.518+0000 7f0511ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.518+0000 7f0511ffb640 1 -- 172.21.15.110:0/2539753547 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0504001cb0 con 0x7f05140a98f0 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.518+0000 7f0519625640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.518+0000 7f0519625640 4 set_mon_vals no callback set 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.518+0000 7f0511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.518+0000 7f0511ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.518+0000 7f0511ffb640 10 monclient: dump: 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.518 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:12.519 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:12.519 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:12.519 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:12.519 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:12.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.519+0000 7f0519625640 10 monclient: shutdown 2024-03-25T19:47:12.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.519+0000 7f0519625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:12.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.519+0000 7f0519625640 1 -- 172.21.15.110:0/2539753547 >> v1:172.21.15.178:6789/0 conn(0x7f05140a98f0 legacy=0x7f05140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.519+0000 7f0519625640 1 -- 172.21.15.110:0/2539753547 shutdown_connections 2024-03-25T19:47:12.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.519+0000 7f0519625640 1 -- 172.21.15.110:0/2539753547 >> 172.21.15.110:0/2539753547 conn(0x7f05140b7ea0 msgr2=0x7f05140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:12.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.519+0000 7f0519625640 1 -- 172.21.15.110:0/2539753547 shutdown_connections 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.519+0000 7f0519625640 1 -- 172.21.15.110:0/2539753547 wait complete. 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 10 monclient: build_initial_monmap 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 10 monclient: monmap: 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 1 librados: starting msgr at 2024-03-25T19:47:12.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 1 librados: starting objecter 2024-03-25T19:47:12.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 1 Processor -- start 2024-03-25T19:47:12.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 1 -- start start 2024-03-25T19:47:12.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 1 librados: setting wanted keys 2024-03-25T19:47:12.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 1 librados: calling monclient init 2024-03-25T19:47:12.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.520+0000 7f0519625640 10 monclient: init 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 10 monclient(hunting): picked mon.noname-c con 0x7f05140a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 10 monclient(hunting): picked mon.noname-a con 0x7f051413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 10 monclient(hunting): picked mon.noname-b con 0x7f05141430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0514139a30 con 0x7f051413f710 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05141d7490 con 0x7f05141430c0 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05141d8360 con 0x7f05140a98f0 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:12.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.521+0000 7f0519625640 10 monclient(hunting): authenticate will time out at 4603.662377s 2024-03-25T19:47:12.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.539+0000 7f05137fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f05141430c0 0x7f0514134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:48006/0 (socket says 172.21.15.110:48006) 2024-03-25T19:47:12.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.539+0000 7f05137fe640 1 -- 172.21.15.110:0/2382954372 learned_addr learned my addr 172.21.15.110:0/2382954372 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:12.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.540+0000 7f04fb7fe640 10 client.?.objecter ms_handle_connect 0x7f051413f710 2024-03-25T19:47:12.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.540+0000 7f04fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:12.544 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.544+0000 7f04fb7fe640 10 client.?.objecter ms_handle_connect 0x7f05141430c0 2024-03-25T19:47:12.544 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.544+0000 7f04fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:12.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.547+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682871028 0 0) 0x7f0514139a30 con 0x7f051413f710 2024-03-25T19:47:12.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.547+0000 7f04fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.547+0000 7f04fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.547+0000 7f04fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.547+0000 7f04fb7fe640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:47:12.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.547+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0500004eb0 con 0x7f051413f710 2024-03-25T19:47:12.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.549+0000 7f04fb7fe640 10 client.?.objecter ms_handle_connect 0x7f05140a98f0 2024-03-25T19:47:12.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.549+0000 7f04fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.559+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347834414 0 0) 0x7f05141d8360 con 0x7f05140a98f0 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.559+0000 7f04fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.559+0000 7f04fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.559+0000 7f04fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.559+0000 7f04fb7fe640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.559+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0514139a30 con 0x7f05140a98f0 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.560+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339716604 0 0) 0x7f05141d7490 con 0x7f05141430c0 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.560+0000 7f04fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.560+0000 7f04fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.560+0000 7f04fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.560+0000 7f04fb7fe640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:47:12.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.560+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05141d8360 con 0x7f05141430c0 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.580+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 82311573 0 0) 0x7f0500004eb0 con 0x7f051413f710 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.580+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05141d7490 con 0x7f051413f710 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.580+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0508001930 con 0x7f051413f710 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.580+0000 7f04fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.580+0000 7f04fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.580+0000 7f04fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:12.580 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.581 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.581 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.581 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:12.581 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:12.581 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:12.581 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:12.581 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:12.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.580+0000 7f04fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:12.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.588+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1428598055 0 0) 0x7f0514139a30 con 0x7f05140a98f0 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.588+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0500004eb0 con 0x7f05140a98f0 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.588+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0504001b70 con 0x7f05140a98f0 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.588+0000 7f04fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.588+0000 7f04fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.588+0000 7f04fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:12.588 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:12.589 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:12.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4294571529 0 0) 0x7f05141d7490 con 0x7f051413f710 2024-03-25T19:47:12.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 >> v1:172.21.15.110:6790/0 conn(0x7f05140a98f0 legacy=0x7f0514138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 >> v1:172.21.15.178:6789/0 conn(0x7f05141430c0 legacy=0x7f0514134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 1 monclient: found mon.a 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05141d7730 con 0x7f051413f710 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f04fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 5 monclient: authenticate success, global_id 4524 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 10 monclient: _renew_subs 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0514134720 con 0x7f051413f710 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 10 monclient: _renew_subs 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:12.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0514137a90 con 0x7f051413f710 2024-03-25T19:47:12.592 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.589+0000 7f0519625640 1 librados: init done 2024-03-25T19:47:12.592 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.590+0000 7f0519625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:12.592 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.590+0000 7f0519625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:12.592 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.590+0000 7f0519625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:12.592 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.590+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05141d78c0 con 0x7f051413f710 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.841+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05080035f0 con 0x7f051413f710 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.841+0000 7f04fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.841+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0508003980 con 0x7f051413f710 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.841+0000 7f04fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.841+0000 7f04fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.841+0000 7f04fb7fe640 10 monclient: dump: 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:12.841 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.842 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:12.842 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:12.842 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:12.842 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:12.842 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:12.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.841+0000 7f05117fa640 4 set_mon_vals no callback set 2024-03-25T19:47:12.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.841+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f0508003ba0 con 0x7f051413f710 2024-03-25T19:47:12.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.842+0000 7f0512ffd640 10 monclient: get_auth_request con 0x7f050006df70 auth_method 0 2024-03-25T19:47:12.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.842+0000 7f04fb7fe640 10 client.4524.objecter ms_handle_connect 0x7f050006df70 2024-03-25T19:47:12.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.848+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f0508053600 con 0x7f051413f710 2024-03-25T19:47:12.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.848+0000 7f04fb7fe640 10 client.4524.objecter ms_dispatch 0x7f0514000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:12.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.848+0000 7f04fb7fe640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:12.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.848+0000 7f04fb7fe640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:12.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.848+0000 7f04fb7fe640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-25T19:47:12.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.848+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f050804f110 con 0x7f051413f710 2024-03-25T19:47:12.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.848+0000 7f04fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:12.848 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:12.848+0000 7f04fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:13.074 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.074+0000 7f0519625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:13.074 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.074+0000 7f0519625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:13.074 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.074+0000 7f0519625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:13.075 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.074+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05140627b0 con 0x7f051413f710 2024-03-25T19:47:13.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.111+0000 7f04fb7fe640 1 -- 172.21.15.110:0/2382954372 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f0508051040 con 0x7f051413f710 2024-03-25T19:47:13.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.111+0000 7f04fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:13.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.111+0000 7f04fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:13.111 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:47:13.111 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 10 librados: watch_flush enter 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 10 librados: watch_flush exit 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 10 client.4524.objecter successfully canceled tick 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 >> v1:172.21.15.178:6800/1132739058 conn(0x7f050006df70 legacy=0x7f0500070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 10 monclient: shutdown 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 >> v1:172.21.15.110:6789/0 conn(0x7f051413f710 legacy=0x7f05141d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 shutdown_connections 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.113+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 >> 172.21.15.110:0/2382954372 conn(0x7f05140b7ea0 msgr2=0x7f0514074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.114+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 shutdown_connections 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.114+0000 7f0519625640 1 -- 172.21.15.110:0/2382954372 wait complete. 2024-03-25T19:47:13.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:13.114+0000 7f0519625640 1 librados: shutdown 2024-03-25T19:47:13.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:19.932 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.045+0000 7f72970db640 10 monclient: get_monmap_and_config 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.045+0000 7f72970db640 10 monclient: build_initial_monmap 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.045+0000 7f72970db640 10 monclient: monmap: 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:47:20.046868+0000 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:47:20.046868+0000 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:20.046 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:20.047 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:20.047 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:47:20.047 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:47:20.047 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:20.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.047+0000 7f72970db640 1 Processor -- start 2024-03-25T19:47:20.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.047+0000 7f72970db640 1 -- start start 2024-03-25T19:47:20.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.047+0000 7f72970db640 10 monclient: init 2024-03-25T19:47:20.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:20.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:20.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 10 monclient(hunting): picked mon.noname-b con 0x7f72900a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:20.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 10 monclient(hunting): picked mon.noname-c con 0x7f729013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:20.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 10 monclient(hunting): picked mon.noname-a con 0x7f72901430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:20.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7290142520 con 0x7f72901430c0 2024-03-25T19:47:20.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72900b28a0 con 0x7f729013f710 2024-03-25T19:47:20.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72900b2a30 con 0x7f72900a98f0 2024-03-25T19:47:20.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:20.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.048+0000 7f72970db640 10 monclient(hunting): authenticate will time out at 4611.189557s 2024-03-25T19:47:20.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.061+0000 7f728ffff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f729013f710 0x7f72901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:52600/0 (socket says 172.21.15.110:52600) 2024-03-25T19:47:20.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.061+0000 7f728ffff640 1 -- 172.21.15.110:0/2996676420 learned_addr learned my addr 172.21.15.110:0/2996676420 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:20.074 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.074+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323779706 0 0) 0x7f72900b28a0 con 0x7f729013f710 2024-03-25T19:47:20.074 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.074+0000 7f728f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:20.074 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.074+0000 7f728f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:20.074 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.074+0000 7f728f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:20.074 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.074+0000 7f728f7fe640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:47:20.075 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.074+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7280003520 con 0x7f729013f710 2024-03-25T19:47:20.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.076+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820303549 0 0) 0x7f72900b2a30 con 0x7f72900a98f0 2024-03-25T19:47:20.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.076+0000 7f728f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:20.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.076+0000 7f728f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:20.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.076+0000 7f728f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:20.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.076+0000 7f728f7fe640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:47:20.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.076+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72900b28a0 con 0x7f72900a98f0 2024-03-25T19:47:20.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1912958356 0 0) 0x7f7280003520 con 0x7f729013f710 2024-03-25T19:47:20.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 >> v1:172.21.15.110:6790/0 conn(0x7f72900a98f0 legacy=0x7f72900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 >> v1:172.21.15.110:6789/0 conn(0x7f72901430c0 legacy=0x7f729014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:20.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:47:20.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:20.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900b44c0 con 0x7f729013f710 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f727c001690 con 0x7f729013f710 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f728f7fe640 10 monclient: dump: 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.092 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:20.093 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:20.093 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:20.093 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:20.093 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:20.093 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:20.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f72970db640 5 monclient: authenticate success, global_id 4540 2024-03-25T19:47:20.093 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.091+0000 7f72970db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:20.096 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.096+0000 7f728f7fe640 1 -- 172.21.15.110:0/2996676420 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f727c0017f0 con 0x7f729013f710 2024-03-25T19:47:20.096 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.096+0000 7f728f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:20.096 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.096+0000 7f72970db640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:20.096 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.096+0000 7f72970db640 4 set_mon_vals no callback set 2024-03-25T19:47:20.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.097+0000 7f72970db640 10 monclient: shutdown 2024-03-25T19:47:20.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.097+0000 7f72970db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:20.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.097+0000 7f72970db640 1 -- 172.21.15.110:0/2996676420 >> v1:172.21.15.178:6789/0 conn(0x7f729013f710 legacy=0x7f72901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.097+0000 7f72970db640 1 -- 172.21.15.110:0/2996676420 shutdown_connections 2024-03-25T19:47:20.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.097+0000 7f72970db640 1 -- 172.21.15.110:0/2996676420 >> 172.21.15.110:0/2996676420 conn(0x7f72900b7ea0 msgr2=0x7f72900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:20.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.097+0000 7f72970db640 1 -- 172.21.15.110:0/2996676420 shutdown_connections 2024-03-25T19:47:20.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.097+0000 7f72970db640 1 -- 172.21.15.110:0/2996676420 wait complete. 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.098+0000 7f72970db640 10 monclient: build_initial_monmap 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.098+0000 7f72970db640 10 monclient: monmap: 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 1 librados: starting msgr at 2024-03-25T19:47:20.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 1 librados: starting objecter 2024-03-25T19:47:20.100 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 1 Processor -- start 2024-03-25T19:47:20.100 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 1 -- start start 2024-03-25T19:47:20.100 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 1 librados: setting wanted keys 2024-03-25T19:47:20.100 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 1 librados: calling monclient init 2024-03-25T19:47:20.100 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 10 monclient: init 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 10 monclient(hunting): picked mon.noname-a con 0x7f72900a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.099+0000 7f72970db640 10 monclient(hunting): picked mon.noname-c con 0x7f729013f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.100+0000 7f72970db640 10 monclient(hunting): picked mon.noname-b con 0x7f72901430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.100+0000 7f72970db640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7290136a10 con 0x7f72900a98f0 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.100+0000 7f72970db640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7290133b60 con 0x7f72901430c0 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.100+0000 7f72970db640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72900ab950 con 0x7f729013f710 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.100+0000 7f72970db640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:20.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.100+0000 7f72970db640 10 monclient(hunting): authenticate will time out at 4611.240877s 2024-03-25T19:47:20.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.111+0000 7f728ffff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f729013f710 0x7f7290136300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:40036/0 (socket says 172.21.15.110:40036) 2024-03-25T19:47:20.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.111+0000 7f728ffff640 1 -- 172.21.15.110:0/3300038486 learned_addr learned my addr 172.21.15.110:0/3300038486 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:20.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.112+0000 7f728d7fa640 10 client.?.objecter ms_handle_connect 0x7f729013f710 2024-03-25T19:47:20.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.112+0000 7f728d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:20.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.119+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524555527 0 0) 0x7f72900ab950 con 0x7f729013f710 2024-03-25T19:47:20.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.119+0000 7f728d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:20.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.119+0000 7f728d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:20.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.119+0000 7f728d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:20.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.119+0000 7f728d7fa640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:47:20.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.119+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7274004eb0 con 0x7f729013f710 2024-03-25T19:47:20.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.123+0000 7f728d7fa640 10 client.?.objecter ms_handle_connect 0x7f72900a98f0 2024-03-25T19:47:20.123 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.123+0000 7f728d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:20.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.125+0000 7f728d7fa640 10 client.?.objecter ms_handle_connect 0x7f72901430c0 2024-03-25T19:47:20.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.125+0000 7f728d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:20.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.130+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954357236 0 0) 0x7f7290136a10 con 0x7f72900a98f0 2024-03-25T19:47:20.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.130+0000 7f728d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:20.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.130+0000 7f728d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:20.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.130+0000 7f728d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:20.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.130+0000 7f728d7fa640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:47:20.130 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.130+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72900ab950 con 0x7f72900a98f0 2024-03-25T19:47:20.139 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1508623856 0 0) 0x7f7274004eb0 con 0x7f729013f710 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7290136a10 con 0x7f729013f710 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f727c001d60 con 0x7f729013f710 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 10 monclient(hunting): dump: 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977065933 0 0) 0x7f7290133b60 con 0x7f72901430c0 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.139+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7274004eb0 con 0x7f72901430c0 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.140+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2315854981 0 0) 0x7f72900ab950 con 0x7f72900a98f0 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.140+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7290133b60 con 0x7f72900a98f0 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.140+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f7284001930 con 0x7f72900a98f0 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.140+0000 7f728d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.140+0000 7f728d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.140+0000 7f728d7fa640 10 monclient(hunting): dump: 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:20.141 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:20.142 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.142 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.142 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:20.142 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:20.142 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:20.142 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:20.142 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:20.142 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:20.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148856569 0 0) 0x7f7274004eb0 con 0x7f72901430c0 2024-03-25T19:47:20.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72900ab950 con 0x7f72901430c0 2024-03-25T19:47:20.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f7288001a60 con 0x7f72901430c0 2024-03-25T19:47:20.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:20.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 10 monclient(hunting): dump: 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2784764374 0 0) 0x7f7290136a10 con 0x7f729013f710 2024-03-25T19:47:20.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 >> v1:172.21.15.178:6789/0 conn(0x7f72901430c0 legacy=0x7f7290133450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 >> v1:172.21.15.110:6789/0 conn(0x7f72900a98f0 legacy=0x7f7290132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 1 monclient: found mon.c 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900abc30 con 0x7f729013f710 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f728d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f72970db640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f72970db640 10 monclient: _renew_subs 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f72970db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.148+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72900abf10 con 0x7f729013f710 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 10 monclient: _renew_subs 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:20.151 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7290137e00 con 0x7f729013f710 2024-03-25T19:47:20.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 1 librados: init done 2024-03-25T19:47:20.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:20.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:20.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:20.152 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.149+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72900627b0 con 0x7f729013f710 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.439+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f727c004a90 con 0x7f729013f710 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.439+0000 7f728d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.439+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f727c004e20 con 0x7f729013f710 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.439+0000 7f728d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.439+0000 7f728d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.439+0000 7f728d7fa640 10 monclient: dump: 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:20.439 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:20.440 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:20.440 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:20.440 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:20.440 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:20.440 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:20.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.439+0000 7f728effd640 4 set_mon_vals no callback set 2024-03-25T19:47:20.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.439+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f727c005040 con 0x7f729013f710 2024-03-25T19:47:20.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.440+0000 7f7294e77640 10 monclient: get_auth_request con 0x7f727406e160 auth_method 0 2024-03-25T19:47:20.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.440+0000 7f728d7fa640 10 client.4541.objecter ms_handle_connect 0x7f727406e160 2024-03-25T19:47:20.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.446+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f727c054990 con 0x7f729013f710 2024-03-25T19:47:20.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.446+0000 7f728d7fa640 10 client.4541.objecter ms_dispatch 0x7f7290000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:20.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.446+0000 7f728d7fa640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:20.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.446+0000 7f728d7fa640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:20.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.446+0000 7f728d7fa640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-25T19:47:20.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.446+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f727c0504a0 con 0x7f729013f710 2024-03-25T19:47:20.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.446+0000 7f728d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:20.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.446+0000 7f728d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:20.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.698+0000 7f72970db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.698+0000 7f72970db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.698+0000 7f72970db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:20.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.698+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f729013c060 con 0x7f729013f710 2024-03-25T19:47:20.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.727+0000 7f728d7fa640 1 -- 172.21.15.110:0/3300038486 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f727c0523d0 con 0x7f729013f710 2024-03-25T19:47:20.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.727+0000 7f728d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.727+0000 7f728d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:20.727 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:47:20.727 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 10 librados: watch_flush enter 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 10 librados: watch_flush exit 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 10 client.4541.objecter successfully canceled tick 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 >> v1:172.21.15.178:6800/1132739058 conn(0x7f727406e160 legacy=0x7f7274070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 10 monclient: shutdown 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 >> v1:172.21.15.110:6790/0 conn(0x7f729013f710 legacy=0x7f7290136300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 shutdown_connections 2024-03-25T19:47:20.730 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.729+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 >> 172.21.15.110:0/3300038486 conn(0x7f72900b7ea0 msgr2=0x7f7290074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:20.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.730+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 shutdown_connections 2024-03-25T19:47:20.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.730+0000 7f72970db640 1 -- 172.21.15.110:0/3300038486 wait complete. 2024-03-25T19:47:20.731 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:20.730+0000 7f72970db640 1 librados: shutdown 2024-03-25T19:47:20.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:23.248 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:30.532 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:23.297 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:30.581 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:23.324 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:30.608 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:23.361 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:30.615 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:23.366 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:30.621 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:23.369 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:30.653 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:23.375 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:30.660 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:23.405 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:30.660 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:23.420 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:30.704 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:27.548 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:27.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.663+0000 7f2db64a8640 10 monclient: get_monmap_and_config 2024-03-25T19:47:27.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.663+0000 7f2db64a8640 10 monclient: build_initial_monmap 2024-03-25T19:47:27.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.663+0000 7f2db64a8640 10 monclient: monmap: 2024-03-25T19:47:27.663 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:27.663 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:27.663 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:47:27.664676+0000 2024-03-25T19:47:27.664 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:47:27.664676+0000 2024-03-25T19:47:27.664 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:27.664 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:27.664 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:27.664 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:47:27.664 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:47:27.664 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:27.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.664+0000 7f2db64a8640 1 Processor -- start 2024-03-25T19:47:27.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.664+0000 7f2db64a8640 1 -- start start 2024-03-25T19:47:27.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.664+0000 7f2db64a8640 10 monclient: init 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2db00a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2db013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2db01430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db0142520 con 0x7f2db013f710 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db00b28a0 con 0x7f2db00a98f0 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db00b2a30 con 0x7f2db01430c0 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db64a8640 10 monclient(hunting): authenticate will time out at 4618.806018s 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db4a45640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f2db01430c0 0x7f2db014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:35406/0 (socket says 172.21.15.110:35406) 2024-03-25T19:47:27.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.665+0000 7f2db4a45640 1 -- 172.21.15.110:0/3018885912 learned_addr learned my addr 172.21.15.110:0/3018885912 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:27.763 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.764+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106559783 0 0) 0x7f2db0142520 con 0x7f2db013f710 2024-03-25T19:47:27.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.764+0000 7f2daeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.764+0000 7f2daeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.764+0000 7f2daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.764+0000 7f2daeffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:47:27.764 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.764+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d98003520 con 0x7f2db013f710 2024-03-25T19:47:27.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.772+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079477820 0 0) 0x7f2db00b28a0 con 0x7f2db00a98f0 2024-03-25T19:47:27.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.772+0000 7f2daeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.772+0000 7f2daeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.772+0000 7f2daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.772+0000 7f2daeffd640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:47:27.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.772+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0142520 con 0x7f2db00a98f0 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1586111487 0 0) 0x7f2d98003520 con 0x7f2db013f710 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 >> v1:172.21.15.110:6790/0 conn(0x7f2db01430c0 legacy=0x7f2db014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 >> v1:172.21.15.178:6789/0 conn(0x7f2db00a98f0 legacy=0x7f2db00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 1 monclient: found mon.noname-a 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db00b44c0 con 0x7f2db013f710 2024-03-25T19:47:27.779 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2da4001690 con 0x7f2db013f710 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2db64a8640 5 monclient: authenticate success, global_id 4533 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2db64a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2daeffd640 10 monclient: dump: 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:27.780 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:27.781 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:27.781 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:27.781 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.779+0000 7f2db64a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:27.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.792+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2da40017f0 con 0x7f2db013f710 2024-03-25T19:47:27.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2daeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:27.792 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2daeffd640 1 -- 172.21.15.110:0/3018885912 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2da4001c00 con 0x7f2db013f710 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2db64a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2db64a8640 4 set_mon_vals no callback set 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2daeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2daeffd640 10 monclient: dump: 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:27.793 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:27.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2db64a8640 10 monclient: shutdown 2024-03-25T19:47:27.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2db64a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:27.794 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2db64a8640 1 -- 172.21.15.110:0/3018885912 >> v1:172.21.15.110:6789/0 conn(0x7f2db013f710 legacy=0x7f2db01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2db64a8640 1 -- 172.21.15.110:0/3018885912 shutdown_connections 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2db64a8640 1 -- 172.21.15.110:0/3018885912 >> 172.21.15.110:0/3018885912 conn(0x7f2db00b7ea0 msgr2=0x7f2db00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.793+0000 7f2db64a8640 1 -- 172.21.15.110:0/3018885912 shutdown_connections 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.794+0000 7f2db64a8640 1 -- 172.21.15.110:0/3018885912 wait complete. 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.794+0000 7f2db64a8640 10 monclient: build_initial_monmap 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.794+0000 7f2db64a8640 10 monclient: monmap: 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:47:27.795 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:27.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.795+0000 7f2db64a8640 1 librados: starting msgr at 2024-03-25T19:47:27.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.795+0000 7f2db64a8640 1 librados: starting objecter 2024-03-25T19:47:27.796 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.795+0000 7f2db64a8640 1 Processor -- start 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.795+0000 7f2db64a8640 1 -- start start 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.795+0000 7f2db64a8640 1 librados: setting wanted keys 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.795+0000 7f2db64a8640 1 librados: calling monclient init 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.795+0000 7f2db64a8640 10 monclient: init 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2db00a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2db013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2db01430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db0137080 con 0x7f2db013f710 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db0137360 con 0x7f2db00a98f0 2024-03-25T19:47:27.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db0135b10 con 0x7f2db01430c0 2024-03-25T19:47:27.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:27.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.796+0000 7f2db64a8640 10 monclient(hunting): authenticate will time out at 4618.937213s 2024-03-25T19:47:27.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.803+0000 7f2daffff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f2db00a98f0 0x7f2db0137a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:53774/0 (socket says 172.21.15.110:53774) 2024-03-25T19:47:27.802 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.803+0000 7f2daffff640 1 -- 172.21.15.110:0/715589929 learned_addr learned my addr 172.21.15.110:0/715589929 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:27.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.803+0000 7f2dacff9640 10 client.?.objecter ms_handle_connect 0x7f2db00a98f0 2024-03-25T19:47:27.803 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.803+0000 7f2dacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.807+0000 7f2dacff9640 10 client.?.objecter ms_handle_connect 0x7f2db013f710 2024-03-25T19:47:27.807 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.807+0000 7f2dacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.810+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228180231 0 0) 0x7f2db0137080 con 0x7f2db013f710 2024-03-25T19:47:27.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.810+0000 7f2dacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.810+0000 7f2dacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.810+0000 7f2dacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.810+0000 7f2dacff9640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:47:27.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.810+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d9c004eb0 con 0x7f2db013f710 2024-03-25T19:47:27.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.811+0000 7f2dacff9640 10 client.?.objecter ms_handle_connect 0x7f2db01430c0 2024-03-25T19:47:27.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.811+0000 7f2dacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.811+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037331643 0 0) 0x7f2db0137360 con 0x7f2db00a98f0 2024-03-25T19:47:27.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.811+0000 7f2dacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.811+0000 7f2dacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.811+0000 7f2dacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.811+0000 7f2dacff9640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:47:27.811 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.812+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0137080 con 0x7f2db00a98f0 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.820+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 475326500 0 0) 0x7f2db0137080 con 0x7f2db00a98f0 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.820+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2db0137360 con 0x7f2db00a98f0 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.820+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2da00019f0 con 0x7f2db00a98f0 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.820+0000 7f2dacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.820+0000 7f2dacff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.820+0000 7f2dacff9640 10 monclient(hunting): dump: 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:27.820 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:27.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.820+0000 7f2dacff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:27.824 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.824+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 588380412 0 0) 0x7f2d9c004eb0 con 0x7f2db013f710 2024-03-25T19:47:27.824 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.824+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2db0137080 con 0x7f2db013f710 2024-03-25T19:47:27.824 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.824+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2da4001bc0 con 0x7f2db013f710 2024-03-25T19:47:27.824 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.824+0000 7f2dacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.824 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.824+0000 7f2dacff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.824+0000 7f2dacff9640 10 monclient(hunting): dump: 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:27.825 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 715865656 0 0) 0x7f2db0137080 con 0x7f2db013f710 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 >> v1:172.21.15.110:6790/0 conn(0x7f2db01430c0 legacy=0x7f2db0130340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 >> v1:172.21.15.178:6789/0 conn(0x7f2db00a98f0 legacy=0x7f2db0137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 1 monclient: found mon.a 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0135df0 con 0x7f2db013f710 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2dacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:27.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 5 monclient: authenticate success, global_id 4536 2024-03-25T19:47:27.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 10 monclient: _renew_subs 2024-03-25T19:47:27.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:27.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db0136340 con 0x7f2db013f710 2024-03-25T19:47:27.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:27.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 10 monclient: _renew_subs 2024-03-25T19:47:27.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:27.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db01352e0 con 0x7f2db013f710 2024-03-25T19:47:27.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.836+0000 7f2db64a8640 1 librados: init done 2024-03-25T19:47:27.838 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.837+0000 7f2db64a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:27.838 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.837+0000 7f2db64a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:27.838 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.837+0000 7f2db64a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:27.838 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:27.837+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db00627b0 con 0x7f2db013f710 2024-03-25T19:47:28.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.143+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2da40038a0 con 0x7f2db013f710 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.143+0000 7f2dacff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.143+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2da4003bb0 con 0x7f2db013f710 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.143+0000 7f2dacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.143+0000 7f2dacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.143+0000 7f2dacff9640 10 monclient: dump: 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:28.144 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:28.145 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:28.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.143+0000 7f2dae7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:28.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.144+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f2da4019b80 con 0x7f2db013f710 2024-03-25T19:47:28.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.145+0000 7f2daffff640 10 monclient: get_auth_request con 0x7f2d9c06de90 auth_method 0 2024-03-25T19:47:28.145 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.145+0000 7f2dacff9640 10 client.4536.objecter ms_handle_connect 0x7f2d9c06de90 2024-03-25T19:47:28.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.150+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f2da40536f0 con 0x7f2db013f710 2024-03-25T19:47:28.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.150+0000 7f2dacff9640 10 client.4536.objecter ms_dispatch 0x7f2db0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:28.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.150+0000 7f2dacff9640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:28.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.150+0000 7f2dacff9640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:28.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.150+0000 7f2dacff9640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-25T19:47:28.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.150+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2da404f200 con 0x7f2db013f710 2024-03-25T19:47:28.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.150+0000 7f2dacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:28.150 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.150+0000 7f2dacff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:28.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.395+0000 7f2db64a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:28.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.395+0000 7f2db64a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:28.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.395+0000 7f2db64a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:28.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.395+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2db0074cc0 con 0x7f2db013f710 2024-03-25T19:47:28.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.416+0000 7f2dacff9640 1 -- 172.21.15.110:0/715589929 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f2da4051130 con 0x7f2db013f710 2024-03-25T19:47:28.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.416+0000 7f2dacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:28.416 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.416+0000 7f2dacff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:28.416 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:47:28.416 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:28.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 10 librados: watch_flush enter 2024-03-25T19:47:28.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 10 librados: watch_flush exit 2024-03-25T19:47:28.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 10 client.4536.objecter successfully canceled tick 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 >> v1:172.21.15.178:6800/1132739058 conn(0x7f2d9c06de90 legacy=0x7f2d9c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 10 monclient: shutdown 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 >> v1:172.21.15.110:6789/0 conn(0x7f2db013f710 legacy=0x7f2db00b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 shutdown_connections 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 >> 172.21.15.110:0/715589929 conn(0x7f2db00b7ea0 msgr2=0x7f2db0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 shutdown_connections 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 1 -- 172.21.15.110:0/715589929 wait complete. 2024-03-25T19:47:28.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:28.418+0000 7f2db64a8640 1 librados: shutdown 2024-03-25T19:47:28.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:35.236 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:35.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.349+0000 7f8171d7b640 10 monclient: get_monmap_and_config 2024-03-25T19:47:35.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.349+0000 7f8171d7b640 10 monclient: build_initial_monmap 2024-03-25T19:47:35.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.349+0000 7f8171d7b640 10 monclient: monmap: 2024-03-25T19:47:35.349 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:35.349 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:35.349 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:47:35.350445+0000 2024-03-25T19:47:35.349 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:47:35.350445+0000 2024-03-25T19:47:35.349 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:35.350 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:35.350 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:35.350 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:47:35.350 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:47:35.350 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:35.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.350+0000 7f8171d7b640 1 Processor -- start 2024-03-25T19:47:35.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.350+0000 7f8171d7b640 1 -- start start 2024-03-25T19:47:35.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.350+0000 7f8171d7b640 10 monclient: init 2024-03-25T19:47:35.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 10 monclient(hunting): picked mon.noname-b con 0x7f816c0a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 10 monclient(hunting): picked mon.noname-a con 0x7f816c13f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 10 monclient(hunting): picked mon.noname-c con 0x7f816c1430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c142520 con 0x7f816c13f710 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c0b28a0 con 0x7f816c1430c0 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c0b2a30 con 0x7f816c0a98f0 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:35.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.351+0000 7f8171d7b640 10 monclient(hunting): authenticate will time out at 4626.492245s 2024-03-25T19:47:35.361 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.361+0000 7f816bfff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f816c1430c0 0x7f816c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:43108/0 (socket says 172.21.15.110:43108) 2024-03-25T19:47:35.361 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.361+0000 7f816bfff640 1 -- 172.21.15.110:0/1341949797 learned_addr learned my addr 172.21.15.110:0/1341949797 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:35.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.377+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286900427 0 0) 0x7f816c0b28a0 con 0x7f816c1430c0 2024-03-25T19:47:35.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.377+0000 7f816a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.377+0000 7f816a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.377+0000 7f816a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.377+0000 7f816a7fc640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:47:35.377 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.377+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8154003520 con 0x7f816c1430c0 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495226086 0 0) 0x7f816c142520 con 0x7f816c13f710 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c0b28a0 con 0x7f816c13f710 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651783790 0 0) 0x7f816c0b2a30 con 0x7f816c0a98f0 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c142520 con 0x7f816c0a98f0 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1504450817 0 0) 0x7f8154003520 con 0x7f816c1430c0 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 >> v1:172.21.15.110:6790/0 conn(0x7f816c0a98f0 legacy=0x7f816c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 >> v1:172.21.15.110:6789/0 conn(0x7f816c13f710 legacy=0x7f816c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c0b44c0 con 0x7f816c1430c0 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:35.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8164001a10 con 0x7f816c1430c0 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f8171d7b640 5 monclient: authenticate success, global_id 4552 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f8171d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f816a7fc640 10 monclient: dump: 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:35.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.390+0000 7f8171d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:35.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.402+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8164001c60 con 0x7f816c1430c0 2024-03-25T19:47:35.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.402+0000 7f816a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:35.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.402+0000 7f816a7fc640 1 -- 172.21.15.110:0/1341949797 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8164001ff0 con 0x7f816c1430c0 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.402+0000 7f816a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.403+0000 7f816a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.403+0000 7f816a7fc640 10 monclient: dump: 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.403+0000 7f8171d7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:35.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.403+0000 7f8171d7b640 4 set_mon_vals no callback set 2024-03-25T19:47:35.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.403+0000 7f8171d7b640 10 monclient: shutdown 2024-03-25T19:47:35.404 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.403+0000 7f8171d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:35.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.403+0000 7f8171d7b640 1 -- 172.21.15.110:0/1341949797 >> v1:172.21.15.178:6789/0 conn(0x7f816c1430c0 legacy=0x7f816c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.404+0000 7f8171d7b640 1 -- 172.21.15.110:0/1341949797 shutdown_connections 2024-03-25T19:47:35.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.404+0000 7f8171d7b640 1 -- 172.21.15.110:0/1341949797 >> 172.21.15.110:0/1341949797 conn(0x7f816c0b7ea0 msgr2=0x7f816c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:35.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.404+0000 7f8171d7b640 1 -- 172.21.15.110:0/1341949797 shutdown_connections 2024-03-25T19:47:35.405 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.404+0000 7f8171d7b640 1 -- 172.21.15.110:0/1341949797 wait complete. 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.405+0000 7f8171d7b640 10 monclient: build_initial_monmap 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.405+0000 7f8171d7b640 10 monclient: monmap: 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:47:35.406 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:47:35.407 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:35.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.406+0000 7f8171d7b640 1 librados: starting msgr at 2024-03-25T19:47:35.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.406+0000 7f8171d7b640 1 librados: starting objecter 2024-03-25T19:47:35.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.406+0000 7f8171d7b640 1 Processor -- start 2024-03-25T19:47:35.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.406+0000 7f8171d7b640 1 -- start start 2024-03-25T19:47:35.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.406+0000 7f8171d7b640 1 librados: setting wanted keys 2024-03-25T19:47:35.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.406+0000 7f8171d7b640 1 librados: calling monclient init 2024-03-25T19:47:35.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.406+0000 7f8171d7b640 10 monclient: init 2024-03-25T19:47:35.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:35.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:35.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 10 monclient(hunting): picked mon.noname-c con 0x7f816c0a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:35.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 10 monclient(hunting): picked mon.noname-b con 0x7f816c13f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:35.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 10 monclient(hunting): picked mon.noname-a con 0x7f816c1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:35.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c139a20 con 0x7f816c1430c0 2024-03-25T19:47:35.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c137d80 con 0x7f816c13f710 2024-03-25T19:47:35.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c133780 con 0x7f816c0a98f0 2024-03-25T19:47:35.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:35.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.407+0000 7f8171d7b640 10 monclient(hunting): authenticate will time out at 4626.548393s 2024-03-25T19:47:35.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.419+0000 7f816affd640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f816c13f710 0x7f816c137670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:43152/0 (socket says 172.21.15.110:43152) 2024-03-25T19:47:35.418 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.419+0000 7f816affd640 1 -- 172.21.15.110:0/853978236 learned_addr learned my addr 172.21.15.110:0/853978236 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:35.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.419+0000 7f8153fff640 10 client.?.objecter ms_handle_connect 0x7f816c13f710 2024-03-25T19:47:35.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.419+0000 7f8153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:35.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.428+0000 7f8153fff640 10 client.?.objecter ms_handle_connect 0x7f816c1430c0 2024-03-25T19:47:35.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.428+0000 7f8153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:35.428 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.429+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202115385 0 0) 0x7f816c137d80 con 0x7f816c13f710 2024-03-25T19:47:35.429 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.429+0000 7f8153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.429 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.429+0000 7f8153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.429 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.429+0000 7f8153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.429 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.429+0000 7f8153fff640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:47:35.429 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.429+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8158004eb0 con 0x7f816c13f710 2024-03-25T19:47:35.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.431+0000 7f8153fff640 10 client.?.objecter ms_handle_connect 0x7f816c0a98f0 2024-03-25T19:47:35.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.431+0000 7f8153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:35.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.435+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498053728 0 0) 0x7f816c139a20 con 0x7f816c1430c0 2024-03-25T19:47:35.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.435+0000 7f8153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.435+0000 7f8153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.435+0000 7f8153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.435+0000 7f8153fff640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:47:35.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.435+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c137d80 con 0x7f816c1430c0 2024-03-25T19:47:35.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.439+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43740732 0 0) 0x7f816c133780 con 0x7f816c0a98f0 2024-03-25T19:47:35.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.439+0000 7f8153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.439+0000 7f8153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.439+0000 7f8153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.439+0000 7f8153fff640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:47:35.439 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.439+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c139a20 con 0x7f816c0a98f0 2024-03-25T19:47:35.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.446+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1918741207 0 0) 0x7f8158004eb0 con 0x7f816c13f710 2024-03-25T19:47:35.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.446+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f816c133780 con 0x7f816c13f710 2024-03-25T19:47:35.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.446+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81600019b0 con 0x7f816c13f710 2024-03-25T19:47:35.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.446+0000 7f8153fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.446+0000 7f8153fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:35.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.446+0000 7f8153fff640 10 monclient(hunting): dump: 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:35.447 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.446+0000 7f8153fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.447+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4244752858 0 0) 0x7f816c137d80 con 0x7f816c1430c0 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.447+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8158004eb0 con 0x7f816c1430c0 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.447+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8164001f70 con 0x7f816c1430c0 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.447+0000 7f8153fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.447+0000 7f8153fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.447+0000 7f8153fff640 10 monclient(hunting): dump: 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:35.448 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.449 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.449 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.449 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:35.449 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:35.449 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:35.449 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:35.449 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1393170815 0 0) 0x7f8158004eb0 con 0x7f816c1430c0 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 >> v1:172.21.15.110:6790/0 conn(0x7f816c0a98f0 legacy=0x7f816c139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 >> v1:172.21.15.178:6789/0 conn(0x7f816c13f710 legacy=0x7f816c137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 1 monclient: found mon.a 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c133910 con 0x7f816c1430c0 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:35.452 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8153fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8171d7b640 5 monclient: authenticate success, global_id 4542 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8171d7b640 10 monclient: _renew_subs 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8171d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.451+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c1367d0 con 0x7f816c1430c0 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.452+0000 7f8171d7b640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.452+0000 7f8171d7b640 10 monclient: _renew_subs 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.452+0000 7f8171d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.452+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c0b71f0 con 0x7f816c1430c0 2024-03-25T19:47:35.453 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.452+0000 7f8171d7b640 1 librados: init done 2024-03-25T19:47:35.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.453+0000 7f8171d7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:35.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.453+0000 7f8171d7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:35.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.453+0000 7f8171d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:35.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.453+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816c14f320 con 0x7f816c1430c0 2024-03-25T19:47:35.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.760+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8164001460 con 0x7f816c1430c0 2024-03-25T19:47:35.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.760+0000 7f8153fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:35.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.760+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8164004fb0 con 0x7f816c1430c0 2024-03-25T19:47:35.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.760+0000 7f8153fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.760+0000 7f8153fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:35.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.760+0000 7f8153fff640 10 monclient: dump: 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:35.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.760+0000 7f8169ffb640 4 set_mon_vals no callback set 2024-03-25T19:47:35.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.761+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f816401af80 con 0x7f816c1430c0 2024-03-25T19:47:35.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.762+0000 7f816b7fe640 10 monclient: get_auth_request con 0x7f815806df60 auth_method 0 2024-03-25T19:47:35.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.762+0000 7f8153fff640 10 client.4542.objecter ms_handle_connect 0x7f815806df60 2024-03-25T19:47:35.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.766+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f8164054aa0 con 0x7f816c1430c0 2024-03-25T19:47:35.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.766+0000 7f8153fff640 10 client.4542.objecter ms_dispatch 0x7f816c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:35.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.766+0000 7f8153fff640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:35.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.766+0000 7f8153fff640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:35.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.766+0000 7f8153fff640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-25T19:47:35.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.766+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81640505b0 con 0x7f816c1430c0 2024-03-25T19:47:35.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.766+0000 7f8153fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:35.766 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.766+0000 7f8153fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:35.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.990+0000 7f8171d7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:35.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.990+0000 7f8171d7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:35.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.990+0000 7f8171d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:47:35.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:35.990+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f816c133aa0 con 0x7f816c1430c0 2024-03-25T19:47:36.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.015+0000 7f8153fff640 1 -- 172.21.15.110:0/853978236 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f81640524e0 con 0x7f816c1430c0 2024-03-25T19:47:36.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.015+0000 7f8153fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:36.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.015+0000 7f8153fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:36.015 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:47:36.015 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:36.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 10 librados: watch_flush enter 2024-03-25T19:47:36.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 10 librados: watch_flush exit 2024-03-25T19:47:36.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-25T19:47:36.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 10 client.4542.objecter successfully canceled tick 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 >> v1:172.21.15.178:6800/1132739058 conn(0x7f815806df60 legacy=0x7f8158070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 10 monclient: shutdown 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 >> v1:172.21.15.110:6789/0 conn(0x7f816c1430c0 legacy=0x7f816c133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 shutdown_connections 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 >> 172.21.15.110:0/853978236 conn(0x7f816c0b7ea0 msgr2=0x7f816c13ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 shutdown_connections 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 1 -- 172.21.15.110:0/853978236 wait complete. 2024-03-25T19:47:36.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:36.017+0000 7f8171d7b640 1 librados: shutdown 2024-03-25T19:47:36.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:42.836 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:42.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.947+0000 7fe114b7a640 10 monclient: get_monmap_and_config 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.947+0000 7fe114b7a640 10 monclient: build_initial_monmap 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.948+0000 7fe114b7a640 10 monclient: monmap: 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:47:42.948900+0000 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:47:42.948900+0000 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:47:42.948 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:42.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.948+0000 7fe114b7a640 1 Processor -- start 2024-03-25T19:47:42.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.948+0000 7fe114b7a640 1 -- start start 2024-03-25T19:47:42.949 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.948+0000 7fe114b7a640 10 monclient: init 2024-03-25T19:47:42.950 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:42.950 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:42.950 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 10 monclient(hunting): picked mon.noname-b con 0x7fe1100a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:42.950 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 10 monclient(hunting): picked mon.noname-a con 0x7fe11013f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:42.950 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 10 monclient(hunting): picked mon.noname-c con 0x7fe1101430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:42.950 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe110142520 con 0x7fe11013f710 2024-03-25T19:47:42.950 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1100b28a0 con 0x7fe1101430c0 2024-03-25T19:47:42.950 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1100b2a30 con 0x7fe1100a98f0 2024-03-25T19:47:42.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:42.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.950+0000 7fe114b7a640 10 monclient(hunting): authenticate will time out at 4634.090962s 2024-03-25T19:47:42.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.955+0000 7fe10dd9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe11013f710 0x7fe1101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33518/0 (socket says 172.21.15.110:33518) 2024-03-25T19:47:42.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.955+0000 7fe10dd9b640 1 -- 172.21.15.110:0/691407083 learned_addr learned my addr 172.21.15.110:0/691407083 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:42.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.979+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178034247 0 0) 0x7fe1100b28a0 con 0x7fe1101430c0 2024-03-25T19:47:42.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.979+0000 7fe10d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.979+0000 7fe10d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.979+0000 7fe10d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.979+0000 7fe10d59a640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:47:42.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.979+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f8003520 con 0x7fe1101430c0 2024-03-25T19:47:42.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.981+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527978573 0 0) 0x7fe110142520 con 0x7fe11013f710 2024-03-25T19:47:42.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.981+0000 7fe10d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.981+0000 7fe10d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.981+0000 7fe10d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.981+0000 7fe10d59a640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:47:42.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.981+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1100b28a0 con 0x7fe11013f710 2024-03-25T19:47:42.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.991+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729044567 0 0) 0x7fe1100b2a30 con 0x7fe1100a98f0 2024-03-25T19:47:42.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.991+0000 7fe10d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.991+0000 7fe10d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.991+0000 7fe10d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.991+0000 7fe10d59a640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:47:42.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.991+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe110142520 con 0x7fe1100a98f0 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 897308479 0 0) 0x7fe0f8003520 con 0x7fe1101430c0 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 >> v1:172.21.15.110:6790/0 conn(0x7fe1100a98f0 legacy=0x7fe1100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 >> v1:172.21.15.110:6789/0 conn(0x7fe11013f710 legacy=0x7fe1101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 1 monclient: found mon.noname-c 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1100b44c0 con 0x7fe1101430c0 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 10 monclient: _finish_auth 0 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe108001a10 con 0x7fe1101430c0 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.994+0000 7fe10d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.995+0000 7fe10d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.995+0000 7fe10d59a640 10 monclient: dump: 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.995+0000 7fe114b7a640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:47:42.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:42.995+0000 7fe114b7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:43.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.011+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe108001b70 con 0x7fe1101430c0 2024-03-25T19:47:43.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.011+0000 7fe10d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:43.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.011+0000 7fe10d59a640 1 -- 172.21.15.110:0/691407083 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe108001f80 con 0x7fe1101430c0 2024-03-25T19:47:43.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.011+0000 7fe114b7a640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:43.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.011+0000 7fe114b7a640 4 set_mon_vals no callback set 2024-03-25T19:47:43.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.011+0000 7fe10d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:43.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.011+0000 7fe10d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:43.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.011+0000 7fe10d59a640 10 monclient: dump: 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:43.012 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:43.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.012+0000 7fe114b7a640 10 monclient: shutdown 2024-03-25T19:47:43.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.012+0000 7fe114b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:43.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.012+0000 7fe114b7a640 1 -- 172.21.15.110:0/691407083 >> v1:172.21.15.178:6789/0 conn(0x7fe1101430c0 legacy=0x7fe11014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.012+0000 7fe114b7a640 1 -- 172.21.15.110:0/691407083 shutdown_connections 2024-03-25T19:47:43.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.012+0000 7fe114b7a640 1 -- 172.21.15.110:0/691407083 >> 172.21.15.110:0/691407083 conn(0x7fe1100b7ea0 msgr2=0x7fe1100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:43.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.013+0000 7fe114b7a640 1 -- 172.21.15.110:0/691407083 shutdown_connections 2024-03-25T19:47:43.013 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.013+0000 7fe114b7a640 1 -- 172.21.15.110:0/691407083 wait complete. 2024-03-25T19:47:43.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 10 monclient: build_initial_monmap 2024-03-25T19:47:43.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 10 monclient: monmap: 2024-03-25T19:47:43.014 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:43.014 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:43.014 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:47:43.014 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 1 librados: starting msgr at 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 1 librados: starting objecter 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 1 Processor -- start 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 1 -- start start 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 1 librados: setting wanted keys 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 1 librados: calling monclient init 2024-03-25T19:47:43.015 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.014+0000 7fe114b7a640 10 monclient: init 2024-03-25T19:47:43.016 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.015+0000 7fe114b7a640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:43.016 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.015+0000 7fe114b7a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:43.016 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.015+0000 7fe114b7a640 10 monclient(hunting): picked mon.noname-a con 0x7fe1100a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:43.016 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.015+0000 7fe114b7a640 10 monclient(hunting): picked mon.noname-b con 0x7fe11013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:43.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.015+0000 7fe114b7a640 10 monclient(hunting): picked mon.noname-c con 0x7fe1101430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:43.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.016+0000 7fe114b7a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe110137c60 con 0x7fe1100a98f0 2024-03-25T19:47:43.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.016+0000 7fe114b7a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe110139160 con 0x7fe11013f710 2024-03-25T19:47:43.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.016+0000 7fe114b7a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1101380f0 con 0x7fe1101430c0 2024-03-25T19:47:43.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.016+0000 7fe114b7a640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:43.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.016+0000 7fe114b7a640 10 monclient(hunting): authenticate will time out at 4634.156874s 2024-03-25T19:47:43.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.034+0000 7fe10dd9b640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fe11013f710 0x7fe110133070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:60528/0 (socket says 172.21.15.110:60528) 2024-03-25T19:47:43.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.034+0000 7fe10dd9b640 1 -- 172.21.15.110:0/1875354960 learned_addr learned my addr 172.21.15.110:0/1875354960 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:43.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.038+0000 7fe0feffd640 10 client.?.objecter ms_handle_connect 0x7fe11013f710 2024-03-25T19:47:43.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.038+0000 7fe0feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:43.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.040+0000 7fe0feffd640 10 client.?.objecter ms_handle_connect 0x7fe1100a98f0 2024-03-25T19:47:43.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.040+0000 7fe0feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:43.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.040+0000 7fe0feffd640 10 client.?.objecter ms_handle_connect 0x7fe1101430c0 2024-03-25T19:47:43.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.040+0000 7fe0feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:43.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.051+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909883498 0 0) 0x7fe1101380f0 con 0x7fe1101430c0 2024-03-25T19:47:43.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.051+0000 7fe0feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:43.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.051+0000 7fe0feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:43.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.051+0000 7fe0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:43.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.051+0000 7fe0feffd640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:47:43.051 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.051+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0004eb0 con 0x7fe1101430c0 2024-03-25T19:47:43.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506645055 0 0) 0x7fe110139160 con 0x7fe11013f710 2024-03-25T19:47:43.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:43.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:43.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:43.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:47:43.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1101380f0 con 0x7fe11013f710 2024-03-25T19:47:43.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549870972 0 0) 0x7fe110137c60 con 0x7fe1100a98f0 2024-03-25T19:47:43.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:43.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:43.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:43.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:47:43.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.061+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe110139160 con 0x7fe1100a98f0 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.071+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1575144489 0 0) 0x7fe0f0004eb0 con 0x7fe1101430c0 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.071+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe110137c60 con 0x7fe1101430c0 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.071+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe108001e80 con 0x7fe1101430c0 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.071+0000 7fe0feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.071+0000 7fe0feffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.071+0000 7fe0feffd640 10 monclient(hunting): dump: 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:43.071 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:43.072 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:43.072 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:43.072 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.071+0000 7fe0feffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:43.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.077+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 530555101 0 0) 0x7fe1101380f0 con 0x7fe11013f710 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.078+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f0004eb0 con 0x7fe11013f710 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.078+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe104001970 con 0x7fe11013f710 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.078+0000 7fe0feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.078+0000 7fe0feffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.078+0000 7fe0feffd640 10 monclient(hunting): dump: 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:43.078 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:43.079 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:43.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560180826 0 0) 0x7fe110139160 con 0x7fe1100a98f0 2024-03-25T19:47:43.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1101380f0 con 0x7fe1100a98f0 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe100001830 con 0x7fe1100a98f0 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 10 monclient(hunting): dump: 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:43.084 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1575450981 0 0) 0x7fe110137c60 con 0x7fe1101430c0 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 >> v1:172.21.15.178:6789/0 conn(0x7fe11013f710 legacy=0x7fe110133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 >> v1:172.21.15.110:6789/0 conn(0x7fe1100a98f0 legacy=0x7fe1101dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 1 monclient: found mon.c 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1101397f0 con 0x7fe1101430c0 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.084+0000 7fe0feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:43.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 10 monclient: _renew_subs 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1101db280 con 0x7fe1101430c0 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 10 monclient: _renew_subs 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe110133780 con 0x7fe1101430c0 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.085+0000 7fe114b7a640 1 librados: init done 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.086+0000 7fe114b7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.086+0000 7fe114b7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.086+0000 7fe114b7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:43.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.086+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe110139a40 con 0x7fe1101430c0 2024-03-25T19:47:43.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.409+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe108002020 con 0x7fe1101430c0 2024-03-25T19:47:43.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.409+0000 7fe0feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:43.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.409+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fe108003dc0 con 0x7fe1101430c0 2024-03-25T19:47:43.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.409+0000 7fe0feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:43.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.409+0000 7fe0feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:43.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.409+0000 7fe0feffd640 10 monclient: dump: 2024-03-25T19:47:43.409 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:43.409 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:43.410 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.409+0000 7fe10cd99640 4 set_mon_vals no callback set 2024-03-25T19:47:43.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.410+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fe108019d90 con 0x7fe1101430c0 2024-03-25T19:47:43.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.411+0000 7fe10e59c640 10 monclient: get_auth_request con 0x7fe0f006e0b0 auth_method 0 2024-03-25T19:47:43.411 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.411+0000 7fe0feffd640 10 client.4559.objecter ms_handle_connect 0x7fe0f006e0b0 2024-03-25T19:47:43.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.414+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fe1080538b0 con 0x7fe1101430c0 2024-03-25T19:47:43.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.414+0000 7fe0feffd640 10 client.4559.objecter ms_dispatch 0x7fe110000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:43.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.414+0000 7fe0feffd640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:43.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.414+0000 7fe0feffd640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:43.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.414+0000 7fe0feffd640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-25T19:47:43.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.414+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe10804f1f0 con 0x7fe1101430c0 2024-03-25T19:47:43.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.414+0000 7fe0feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:43.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.414+0000 7fe0feffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:43.453 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:50.737 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.466 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:50.750 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.498 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:50.783 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.515 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:50.769 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.515 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:50.799 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.521 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:50.805 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.521 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:50.776 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.548 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:50.832 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.549 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:07:50.803 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.626+0000 7fe114b7a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.626+0000 7fe114b7a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.626+0000 7fe114b7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:47:43.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.626+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe110137e10 con 0x7fe1101430c0 2024-03-25T19:47:43.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.681+0000 7fe0feffd640 1 -- 172.21.15.110:0/1875354960 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fe110137e10 con 0x7fe1101430c0 2024-03-25T19:47:43.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.681+0000 7fe0feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.681+0000 7fe0feffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:43.682 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:47:43.682 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.683+0000 7fe114b7a640 10 librados: watch_flush enter 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 10 librados: watch_flush exit 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 10 client.4559.objecter successfully canceled tick 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 >> v1:172.21.15.178:6800/1132739058 conn(0x7fe0f006e0b0 legacy=0x7fe0f0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 10 monclient: shutdown 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 >> v1:172.21.15.110:6790/0 conn(0x7fe1101430c0 legacy=0x7fe1101e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 shutdown_connections 2024-03-25T19:47:43.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 >> 172.21.15.110:0/1875354960 conn(0x7fe1100b7ea0 msgr2=0x7fe11013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:43.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 shutdown_connections 2024-03-25T19:47:43.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 1 -- 172.21.15.110:0/1875354960 wait complete. 2024-03-25T19:47:43.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:43.684+0000 7fe114b7a640 1 librados: shutdown 2024-03-25T19:47:43.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:50.502 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:50.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.614+0000 7fd6d273a640 10 monclient: get_monmap_and_config 2024-03-25T19:47:50.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.614+0000 7fd6d273a640 10 monclient: build_initial_monmap 2024-03-25T19:47:50.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.614+0000 7fd6d273a640 10 monclient: monmap: 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:47:50.615480+0000 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:47:50.615480+0000 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:47:50.615 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:50.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.615+0000 7fd6d273a640 1 Processor -- start 2024-03-25T19:47:50.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.615+0000 7fd6d273a640 1 -- start start 2024-03-25T19:47:50.616 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.615+0000 7fd6d273a640 10 monclient: init 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 10 monclient(hunting): picked mon.noname-a con 0x7fd6cc0b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 10 monclient(hunting): picked mon.noname-c con 0x7fd6cc0b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 10 monclient(hunting): picked mon.noname-b con 0x7fd6cc14be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0af940 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0b0a70 con 0x7fd6cc0b1730 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0b0d50 con 0x7fd6cc14be90 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:50.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.616+0000 7fd6d273a640 10 monclient(hunting): authenticate will time out at 4641.757358s 2024-03-25T19:47:50.627 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.628+0000 7fd6d0cd7640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fd6cc14be90 0x7fd6cc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:53578/0 (socket says 172.21.15.110:53578) 2024-03-25T19:47:50.627 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.628+0000 7fd6d0cd7640 1 -- 172.21.15.110:0/605859639 learned_addr learned my addr 172.21.15.110:0/605859639 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:50.640 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.640+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008233722 0 0) 0x7fd6cc0b0a70 con 0x7fd6cc0b1730 2024-03-25T19:47:50.640 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.640+0000 7fd6caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.640 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.640+0000 7fd6caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.640 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.640+0000 7fd6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.640+0000 7fd6caffd640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:47:50.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.640+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b4003520 con 0x7fd6cc0b1730 2024-03-25T19:47:50.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.643+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528141875 0 0) 0x7fd6cc0b0d50 con 0x7fd6cc14be90 2024-03-25T19:47:50.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.643+0000 7fd6caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.643+0000 7fd6caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.643+0000 7fd6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.643+0000 7fd6caffd640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:47:50.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.643+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc0b0a70 con 0x7fd6cc14be90 2024-03-25T19:47:50.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.645+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511822677 0 0) 0x7fd6cc0af940 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.645+0000 7fd6caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.645+0000 7fd6caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.645+0000 7fd6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.645+0000 7fd6caffd640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:47:50.645 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.645+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc0b0d50 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1383368385 0 0) 0x7fd6cc0b0a70 con 0x7fd6cc14be90 2024-03-25T19:47:50.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 >> v1:172.21.15.178:6789/0 conn(0x7fd6cc0b1730 legacy=0x7fd6cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 >> v1:172.21.15.110:6789/0 conn(0x7fd6cc0b4d60 legacy=0x7fd6cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:50.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 1 monclient: found mon.noname-b 2024-03-25T19:47:50.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:50.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc0b1030 con 0x7fd6cc14be90 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd6c40019f0 con 0x7fd6cc14be90 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6caffd640 10 monclient: dump: 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.651 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.652 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:50.652 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:50.652 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:50.652 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:50.652 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:50.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6d273a640 5 monclient: authenticate success, global_id 4562 2024-03-25T19:47:50.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.650+0000 7fd6d273a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:50.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.664+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6c4001b50 con 0x7fd6cc14be90 2024-03-25T19:47:50.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.664+0000 7fd6caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:50.664 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.664+0000 7fd6caffd640 1 -- 172.21.15.110:0/605859639 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd6c4001f60 con 0x7fd6cc14be90 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.664+0000 7fd6d273a640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.664+0000 7fd6d273a640 4 set_mon_vals no callback set 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.664+0000 7fd6caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.664+0000 7fd6caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.664+0000 7fd6caffd640 10 monclient: dump: 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:50.665 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:50.666 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:50.666 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:50.666 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.665+0000 7fd6d273a640 10 monclient: shutdown 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.665+0000 7fd6d273a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.665+0000 7fd6d273a640 1 -- 172.21.15.110:0/605859639 >> v1:172.21.15.110:6790/0 conn(0x7fd6cc14be90 legacy=0x7fd6cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.665+0000 7fd6d273a640 1 -- 172.21.15.110:0/605859639 shutdown_connections 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.665+0000 7fd6d273a640 1 -- 172.21.15.110:0/605859639 >> 172.21.15.110:0/605859639 conn(0x7fd6cc0617a0 msgr2=0x7fd6cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.665+0000 7fd6d273a640 1 -- 172.21.15.110:0/605859639 shutdown_connections 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.665+0000 7fd6d273a640 1 -- 172.21.15.110:0/605859639 wait complete. 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.666+0000 7fd6d273a640 10 monclient: build_initial_monmap 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.666+0000 7fd6d273a640 10 monclient: monmap: 2024-03-25T19:47:50.668 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.666+0000 7fd6d273a640 1 librados: starting msgr at 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.667+0000 7fd6d273a640 1 librados: starting objecter 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.667+0000 7fd6d273a640 1 Processor -- start 2024-03-25T19:47:50.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.667+0000 7fd6d273a640 1 -- start start 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.667+0000 7fd6d273a640 1 librados: setting wanted keys 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.667+0000 7fd6d273a640 1 librados: calling monclient init 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.667+0000 7fd6d273a640 10 monclient: init 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 10 monclient(hunting): picked mon.noname-b con 0x7fd6cc0b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 10 monclient(hunting): picked mon.noname-c con 0x7fd6cc0b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 10 monclient(hunting): picked mon.noname-a con 0x7fd6cc14be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc141920 con 0x7fd6cc14be90 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc140380 con 0x7fd6cc0b1730 2024-03-25T19:47:50.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc140510 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:50.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.668+0000 7fd6d273a640 10 monclient(hunting): authenticate will time out at 4641.809158s 2024-03-25T19:47:50.675 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.675+0000 7fd6d0cd7640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd6cc14be90 0x7fd6cc1d7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:38358/0 (socket says 172.21.15.110:38358) 2024-03-25T19:47:50.675 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.675+0000 7fd6d0cd7640 1 -- 172.21.15.110:0/3097367784 learned_addr learned my addr 172.21.15.110:0/3097367784 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:50.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.679+0000 7fd6c8ff9640 10 client.?.objecter ms_handle_connect 0x7fd6cc14be90 2024-03-25T19:47:50.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.679+0000 7fd6c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.679+0000 7fd6c8ff9640 10 client.?.objecter ms_handle_connect 0x7fd6cc0b1730 2024-03-25T19:47:50.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.679+0000 7fd6c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.684+0000 7fd6c8ff9640 10 client.?.objecter ms_handle_connect 0x7fd6cc0b4d60 2024-03-25T19:47:50.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.684+0000 7fd6c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551754961 0 0) 0x7fd6cc141920 con 0x7fd6cc14be90 2024-03-25T19:47:50.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b8004eb0 con 0x7fd6cc14be90 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801282022 0 0) 0x7fd6cc140380 con 0x7fd6cc0b1730 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:47:50.686 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.685+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc141920 con 0x7fd6cc0b1730 2024-03-25T19:47:50.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.693+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237646562 0 0) 0x7fd6cc140510 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.693+0000 7fd6c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.693+0000 7fd6c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.693+0000 7fd6c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.693+0000 7fd6c8ff9640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:47:50.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.693+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6cc140380 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.699+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2394902102 0 0) 0x7fd6cc141920 con 0x7fd6cc0b1730 2024-03-25T19:47:50.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.699+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6b8005620 con 0x7fd6cc0b1730 2024-03-25T19:47:50.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.699+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd6bc001870 con 0x7fd6cc0b1730 2024-03-25T19:47:50.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.699+0000 7fd6c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.699+0000 7fd6c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.699+0000 7fd6c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:50.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.699+0000 7fd6c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:50.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.701+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3617811040 0 0) 0x7fd6b8004eb0 con 0x7fd6cc14be90 2024-03-25T19:47:50.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.701+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6cc141920 con 0x7fd6cc14be90 2024-03-25T19:47:50.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.701+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd6c4001f60 con 0x7fd6cc14be90 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.701+0000 7fd6c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.701+0000 7fd6c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.701+0000 7fd6c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:50.702 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:50.703 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:50.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.707+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3543886352 0 0) 0x7fd6cc140380 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.707+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6b8004eb0 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.707+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd6c0001950 con 0x7fd6cc0b4d60 2024-03-25T19:47:50.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.707+0000 7fd6c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.707+0000 7fd6c8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:50.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.707+0000 7fd6c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:50.707 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:50.708 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:50.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4225191708 0 0) 0x7fd6b8005620 con 0x7fd6cc0b1730 2024-03-25T19:47:50.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 >> v1:172.21.15.110:6790/0 conn(0x7fd6cc0b4d60 legacy=0x7fd6cc132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 >> v1:172.21.15.110:6789/0 conn(0x7fd6cc14be90 legacy=0x7fd6cc1d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:50.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 1 monclient: found mon.b 2024-03-25T19:47:50.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:50.711 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc143050 con 0x7fd6cc0b1730 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 10 monclient: _renew_subs 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc143300 con 0x7fd6cc0b1730 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 10 monclient: _renew_subs 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc1336a0 con 0x7fd6cc0b1730 2024-03-25T19:47:50.712 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.711+0000 7fd6d273a640 1 librados: init done 2024-03-25T19:47:50.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.712+0000 7fd6d273a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.712+0000 7fd6d273a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.712+0000 7fd6d273a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:50.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.712+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc141710 con 0x7fd6cc0b1730 2024-03-25T19:47:50.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.934+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6bc003610 con 0x7fd6cc0b1730 2024-03-25T19:47:50.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.934+0000 7fd6c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:50.935 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.934+0000 7fd6ca7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:50.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.992+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fd6bc003920 con 0x7fd6cc0b1730 2024-03-25T19:47:50.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.992+0000 7fd6c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.992+0000 7fd6c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.992+0000 7fd6c8ff9640 10 monclient: dump: 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:50.992 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:50.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.993+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fd6bc0198f0 con 0x7fd6cc0b1730 2024-03-25T19:47:50.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.993+0000 7fd6cb7fe640 10 monclient: get_auth_request con 0x7fd6b806e310 auth_method 0 2024-03-25T19:47:50.993 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.993+0000 7fd6c8ff9640 10 client.4567.objecter ms_handle_connect 0x7fd6b806e310 2024-03-25T19:47:50.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.995+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fd6bc053600 con 0x7fd6cc0b1730 2024-03-25T19:47:50.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.995+0000 7fd6c8ff9640 10 client.4567.objecter ms_dispatch 0x7fd6cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:50.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.995+0000 7fd6c8ff9640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:50.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.995+0000 7fd6c8ff9640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:50.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.995+0000 7fd6c8ff9640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T19:47:50.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.998+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6bc04ef40 con 0x7fd6cc0b1730 2024-03-25T19:47:50.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.998+0000 7fd6c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:50.998+0000 7fd6c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:51.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.243+0000 7fd6d273a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:51.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.243+0000 7fd6d273a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:51.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.243+0000 7fd6d273a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:51.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.243+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6cc0627b0 con 0x7fd6cc0b1730 2024-03-25T19:47:51.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.278+0000 7fd6c8ff9640 1 -- 172.21.15.110:0/3097367784 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fd6bc051040 con 0x7fd6cc0b1730 2024-03-25T19:47:51.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.278+0000 7fd6c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:51.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.278+0000 7fd6c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:51.278 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:47:51.278 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:51.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 10 librados: watch_flush enter 2024-03-25T19:47:51.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 10 librados: watch_flush exit 2024-03-25T19:47:51.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 10 client.4567.objecter successfully canceled tick 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 >> v1:172.21.15.178:6800/1132739058 conn(0x7fd6b806e310 legacy=0x7fd6b80707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 10 monclient: shutdown 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 >> v1:172.21.15.178:6789/0 conn(0x7fd6cc0b1730 legacy=0x7fd6cc141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 shutdown_connections 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 >> 172.21.15.110:0/3097367784 conn(0x7fd6cc0617a0 msgr2=0x7fd6cc0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 shutdown_connections 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 1 -- 172.21.15.110:0/3097367784 wait complete. 2024-03-25T19:47:51.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:51.280+0000 7fd6d273a640 1 librados: shutdown 2024-03-25T19:47:51.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:58.098 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:58.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.212+0000 7f626e79d640 10 monclient: get_monmap_and_config 2024-03-25T19:47:58.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.212+0000 7f626e79d640 10 monclient: build_initial_monmap 2024-03-25T19:47:58.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.212+0000 7f626e79d640 10 monclient: monmap: 2024-03-25T19:47:58.213 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:58.213 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:58.214 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:47:58.214636+0000 2024-03-25T19:47:58.214 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:47:58.214636+0000 2024-03-25T19:47:58.214 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:58.214 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:58.214 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:58.214 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:47:58.214 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:47:58.214 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:58.215 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.213+0000 7f626e79d640 1 Processor -- start 2024-03-25T19:47:58.215 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.213+0000 7f626e79d640 1 -- start start 2024-03-25T19:47:58.215 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.213+0000 7f626e79d640 10 monclient: init 2024-03-25T19:47:58.215 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 10 monclient(hunting): picked mon.noname-a con 0x7f62680a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 10 monclient(hunting): picked mon.noname-b con 0x7f626813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 10 monclient(hunting): picked mon.noname-c con 0x7f62681430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6268142520 con 0x7f62680a98f0 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62680b28a0 con 0x7f62681430c0 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62680b2a30 con 0x7f626813f710 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:58.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.214+0000 7f626e79d640 10 monclient(hunting): authenticate will time out at 4649.356451s 2024-03-25T19:47:58.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.221+0000 7f62677fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f626813f710 0x7f62681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:57158/0 (socket says 172.21.15.110:57158) 2024-03-25T19:47:58.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.221+0000 7f62677fe640 1 -- 172.21.15.110:0/2502141564 learned_addr learned my addr 172.21.15.110:0/2502141564 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:58.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.234+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049736276 0 0) 0x7f6268142520 con 0x7f62680a98f0 2024-03-25T19:47:58.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.234+0000 7f6266ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.234+0000 7f6266ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.234+0000 7f6266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.234+0000 7f6266ffd640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:47:58.235 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.234+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6250003520 con 0x7f62680a98f0 2024-03-25T19:47:58.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.238+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127733234 0 0) 0x7f62680b28a0 con 0x7f62681430c0 2024-03-25T19:47:58.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.238+0000 7f6266ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.238+0000 7f6266ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.238+0000 7f6266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.238+0000 7f6266ffd640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:47:58.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.238+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6268142520 con 0x7f62681430c0 2024-03-25T19:47:58.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.241+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672698494 0 0) 0x7f62680b2a30 con 0x7f626813f710 2024-03-25T19:47:58.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.241+0000 7f6266ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.241+0000 7f6266ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.241+0000 7f6266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.241+0000 7f6266ffd640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:47:58.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.241+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62680b28a0 con 0x7f626813f710 2024-03-25T19:47:58.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546388027 0 0) 0x7f6250003520 con 0x7f62680a98f0 2024-03-25T19:47:58.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 >> v1:172.21.15.110:6790/0 conn(0x7f626813f710 legacy=0x7f62681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 >> v1:172.21.15.178:6789/0 conn(0x7f62681430c0 legacy=0x7f626814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:58.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 1 monclient: found mon.noname-a 2024-03-25T19:47:58.255 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62680b44c0 con 0x7f62680a98f0 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6258001690 con 0x7f62680a98f0 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f6266ffd640 10 monclient: dump: 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.256 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.257 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.257 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:58.257 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:58.257 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:58.257 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:58.257 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:58.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f626e79d640 5 monclient: authenticate success, global_id 4557 2024-03-25T19:47:58.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.254+0000 7f626e79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.263+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62580017f0 con 0x7f62680a98f0 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.263+0000 7f6266ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.263+0000 7f6266ffd640 1 -- 172.21.15.110:0/2502141564 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6258001c00 con 0x7f62680a98f0 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.263+0000 7f6266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.263+0000 7f6266ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.263+0000 7f6266ffd640 10 monclient: dump: 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.264 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.265 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:58.265 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:58.265 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:58.265 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:58.265 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:58.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.263+0000 7f626e79d640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:58.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.263+0000 7f626e79d640 4 set_mon_vals no callback set 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.264+0000 7f626e79d640 10 monclient: shutdown 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.264+0000 7f626e79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.264+0000 7f626e79d640 1 -- 172.21.15.110:0/2502141564 >> v1:172.21.15.110:6789/0 conn(0x7f62680a98f0 legacy=0x7f62680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.264+0000 7f626e79d640 1 -- 172.21.15.110:0/2502141564 shutdown_connections 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.264+0000 7f626e79d640 1 -- 172.21.15.110:0/2502141564 >> 172.21.15.110:0/2502141564 conn(0x7f62680b7ea0 msgr2=0x7f62680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.265+0000 7f626e79d640 1 -- 172.21.15.110:0/2502141564 shutdown_connections 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.265+0000 7f626e79d640 1 -- 172.21.15.110:0/2502141564 wait complete. 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.265+0000 7f626e79d640 10 monclient: build_initial_monmap 2024-03-25T19:47:58.266 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.265+0000 7f626e79d640 10 monclient: monmap: 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:47:58.267 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:58.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.266+0000 7f626e79d640 1 librados: starting msgr at 2024-03-25T19:47:58.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.266+0000 7f626e79d640 1 librados: starting objecter 2024-03-25T19:47:58.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.266+0000 7f626e79d640 1 Processor -- start 2024-03-25T19:47:58.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.266+0000 7f626e79d640 1 -- start start 2024-03-25T19:47:58.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.266+0000 7f626e79d640 1 librados: setting wanted keys 2024-03-25T19:47:58.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.266+0000 7f626e79d640 1 librados: calling monclient init 2024-03-25T19:47:58.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.266+0000 7f626e79d640 10 monclient: init 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 10 monclient(hunting): picked mon.noname-c con 0x7f62680a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 10 monclient(hunting): picked mon.noname-a con 0x7f626813f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 10 monclient(hunting): picked mon.noname-b con 0x7f62681430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6268138970 con 0x7f626813f710 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62681d7810 con 0x7f62681430c0 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6268139990 con 0x7f62680a98f0 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:58.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.267+0000 7f626e79d640 10 monclient(hunting): authenticate will time out at 4649.409313s 2024-03-25T19:47:58.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.279+0000 7f62677fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f626813f710 0x7f62681d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37800/0 (socket says 172.21.15.110:37800) 2024-03-25T19:47:58.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.279+0000 7f62677fe640 1 -- 172.21.15.110:0/3218229396 learned_addr learned my addr 172.21.15.110:0/3218229396 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:47:58.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.281+0000 7f6264ff9640 10 client.?.objecter ms_handle_connect 0x7f62681430c0 2024-03-25T19:47:58.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.281+0000 7f6264ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:58.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.281+0000 7f6264ff9640 10 client.?.objecter ms_handle_connect 0x7f626813f710 2024-03-25T19:47:58.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.281+0000 7f6264ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:58.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.288+0000 7f6264ff9640 10 client.?.objecter ms_handle_connect 0x7f62680a98f0 2024-03-25T19:47:58.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.288+0000 7f6264ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:58.292 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.292+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287452453 0 0) 0x7f6268138970 con 0x7f626813f710 2024-03-25T19:47:58.292 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.292+0000 7f6264ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.292+0000 7f6264ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.292+0000 7f6264ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.292+0000 7f6264ff9640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:47:58.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.292+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6254004eb0 con 0x7f626813f710 2024-03-25T19:47:58.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.294+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358144128 0 0) 0x7f62681d7810 con 0x7f62681430c0 2024-03-25T19:47:58.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.294+0000 7f6264ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.294+0000 7f6264ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.294+0000 7f6264ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.294+0000 7f6264ff9640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:47:58.295 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.294+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6268138970 con 0x7f62681430c0 2024-03-25T19:47:58.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.305+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553259422 0 0) 0x7f6268139990 con 0x7f62680a98f0 2024-03-25T19:47:58.305 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.305+0000 7f6264ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.306 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.305+0000 7f6264ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.306 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.305+0000 7f6264ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.306 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.305+0000 7f6264ff9640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:47:58.306 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.305+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62681d7810 con 0x7f62680a98f0 2024-03-25T19:47:58.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.315+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3975661714 0 0) 0x7f6268138970 con 0x7f62681430c0 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.315+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6268139990 con 0x7f62681430c0 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.315+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6260001bd0 con 0x7f62681430c0 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.315+0000 7f6264ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.315+0000 7f6264ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.315+0000 7f6264ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:58.316 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:58.317 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:58.317 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:58.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.315+0000 7f6264ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571938846 0 0) 0x7f62681d7810 con 0x7f62680a98f0 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6268138970 con 0x7f62680a98f0 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6258001b00 con 0x7f62680a98f0 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.327 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4241799895 0 0) 0x7f6268139990 con 0x7f62681430c0 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 >> v1:172.21.15.110:6790/0 conn(0x7f62680a98f0 legacy=0x7f6268138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 >> v1:172.21.15.110:6789/0 conn(0x7f626813f710 legacy=0x7f62681d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 1 monclient: found mon.b 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62681d8440 con 0x7f62681430c0 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:58.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f6264ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f626e79d640 5 monclient: authenticate success, global_id 4573 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f626e79d640 10 monclient: _renew_subs 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f626e79d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.325+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62681d41e0 con 0x7f62681430c0 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.326+0000 7f626e79d640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.326+0000 7f626e79d640 10 monclient: _renew_subs 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.326+0000 7f626e79d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.326+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6268138d80 con 0x7f62681430c0 2024-03-25T19:47:58.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.326+0000 7f626e79d640 1 librados: init done 2024-03-25T19:47:58.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.326+0000 7f626e79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:58.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.326+0000 7f626e79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:58.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.327+0000 7f626e79d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:58.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.327+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62681d79a0 con 0x7f62681430c0 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.604+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6260002060 con 0x7f62681430c0 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.604+0000 7f6264ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.604+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6260003b10 con 0x7f62681430c0 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.604+0000 7f6264ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.604+0000 7f6264ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.604+0000 7f6264ff9640 10 monclient: dump: 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:47:58.605 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.606 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:47:58.606 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.606 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:47:58.606 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:47:58.606 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:47:58.606 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:47:58.606 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:47:58.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.604+0000 7f62667fc640 4 set_mon_vals no callback set 2024-03-25T19:47:58.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.605+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f6260019ae0 con 0x7f62681430c0 2024-03-25T19:47:58.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.606+0000 7f6267fff640 10 monclient: get_auth_request con 0x7f625406de60 auth_method 0 2024-03-25T19:47:58.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.606+0000 7f6264ff9640 10 client.4573.objecter ms_handle_connect 0x7f625406de60 2024-03-25T19:47:58.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.611+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f62600537d0 con 0x7f62681430c0 2024-03-25T19:47:58.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.611+0000 7f6264ff9640 10 client.4573.objecter ms_dispatch 0x7f6268000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:58.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.611+0000 7f6264ff9640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:58.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.611+0000 7f6264ff9640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:58.612 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.611+0000 7f6264ff9640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-25T19:47:58.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.612+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f626004f110 con 0x7f62681430c0 2024-03-25T19:47:58.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.612+0000 7f6264ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:58.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.612+0000 7f6264ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:58.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.869+0000 7f626e79d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.869+0000 7f626e79d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.869+0000 7f626e79d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:47:58.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.869+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62680627b0 con 0x7f62681430c0 2024-03-25T19:47:58.898 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.898+0000 7f6264ff9640 1 -- 172.21.15.110:0/3218229396 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f6260051210 con 0x7f62681430c0 2024-03-25T19:47:58.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.898+0000 7f6264ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.898+0000 7f6264ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:58.899 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:47:58.899 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:58.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 10 librados: watch_flush enter 2024-03-25T19:47:58.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 10 librados: watch_flush exit 2024-03-25T19:47:58.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-25T19:47:58.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 10 client.4573.objecter successfully canceled tick 2024-03-25T19:47:58.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 >> v1:172.21.15.178:6800/1132739058 conn(0x7f625406de60 legacy=0x7f6254070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 10 monclient: shutdown 2024-03-25T19:47:58.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:58.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 >> v1:172.21.15.178:6789/0 conn(0x7f62681430c0 legacy=0x7f62681d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 shutdown_connections 2024-03-25T19:47:58.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 >> 172.21.15.110:0/3218229396 conn(0x7f62680b7ea0 msgr2=0x7f626813c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:58.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 shutdown_connections 2024-03-25T19:47:58.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 1 -- 172.21.15.110:0/3218229396 wait complete. 2024-03-25T19:47:58.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:47:58.900+0000 7f626e79d640 1 librados: shutdown 2024-03-25T19:47:58.912 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:03.432 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:10.716 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:03.444 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:10.728 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:03.478 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:10.732 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:03.485 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:10.769 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:03.499 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:10.753 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:03.509 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:10.793 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:03.513 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:10.797 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:03.528 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:10.783 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:03.547 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:10.831 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.720 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.834+0000 7f112fc86640 10 monclient: get_monmap_and_config 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.834+0000 7f112fc86640 10 monclient: build_initial_monmap 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.834+0000 7f112fc86640 10 monclient: monmap: 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:48:05.835423+0000 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:48:05.835423+0000 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:05.834 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:05.835 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:48:05.835 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:48:05.835 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:05.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.835+0000 7f112fc86640 1 Processor -- start 2024-03-25T19:48:05.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.835+0000 7f112fc86640 1 -- start start 2024-03-25T19:48:05.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.835+0000 7f112fc86640 10 monclient: init 2024-03-25T19:48:05.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:05.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:05.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 10 monclient(hunting): picked mon.noname-c con 0x7f11280b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:05.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 10 monclient(hunting): picked mon.noname-a con 0x7f11280b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:05.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 10 monclient(hunting): picked mon.noname-b con 0x7f112814be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:05.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11280af940 con 0x7f11280b1730 2024-03-25T19:48:05.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11280b0a70 con 0x7f11280b4d60 2024-03-25T19:48:05.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11280b0d50 con 0x7f112814be90 2024-03-25T19:48:05.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:05.837 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.836+0000 7f112fc86640 10 monclient(hunting): authenticate will time out at 4656.977512s 2024-03-25T19:48:05.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.839+0000 7f112e223640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f112814be90 0x7f112814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:45860/0 (socket says 172.21.15.110:45860) 2024-03-25T19:48:05.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.839+0000 7f112e223640 1 -- 172.21.15.110:0/4147474522 learned_addr learned my addr 172.21.15.110:0/4147474522 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:05.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.845+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177966287 0 0) 0x7f11280b0d50 con 0x7f112814be90 2024-03-25T19:48:05.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.845+0000 7f112ca20640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.845+0000 7f112ca20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.845+0000 7f112ca20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.845+0000 7f112ca20640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:48:05.846 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.845+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1114003520 con 0x7f112814be90 2024-03-25T19:48:05.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.853+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293419469 0 0) 0x7f11280af940 con 0x7f11280b1730 2024-03-25T19:48:05.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.853+0000 7f112ca20640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.853+0000 7f112ca20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.853+0000 7f112ca20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.853+0000 7f112ca20640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:48:05.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.853+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11280b0d50 con 0x7f11280b1730 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2646840809 0 0) 0x7f1114003520 con 0x7f112814be90 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 >> v1:172.21.15.178:6789/0 conn(0x7f11280b4d60 legacy=0x7f112813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 >> v1:172.21.15.110:6789/0 conn(0x7f11280b1730 legacy=0x7f11280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 10 monclient: _finish_hunting 0 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 1 monclient: found mon.noname-b 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11280b1030 con 0x7f112814be90 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 10 monclient: _finish_auth 0 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 10 monclient: _check_auth_tickets 2024-03-25T19:48:05.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1120001a10 con 0x7f112814be90 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112ca20640 10 monclient: dump: 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112fc86640 5 monclient: authenticate success, global_id 4574 2024-03-25T19:48:05.859 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.858+0000 7f112fc86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.870+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1120001b70 con 0x7f112814be90 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.870+0000 7f112ca20640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.870+0000 7f112ca20640 1 -- 172.21.15.110:0/4147474522 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1120001f80 con 0x7f112814be90 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.870+0000 7f112fc86640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.870+0000 7f112fc86640 4 set_mon_vals no callback set 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.870+0000 7f112ca20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.870+0000 7f112ca20640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.870+0000 7f112ca20640 10 monclient: dump: 2024-03-25T19:48:05.870 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:05.871 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:05.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 10 monclient: shutdown 2024-03-25T19:48:05.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:05.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 1 -- 172.21.15.110:0/4147474522 >> v1:172.21.15.110:6790/0 conn(0x7f112814be90 legacy=0x7f112814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 1 -- 172.21.15.110:0/4147474522 shutdown_connections 2024-03-25T19:48:05.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 1 -- 172.21.15.110:0/4147474522 >> 172.21.15.110:0/4147474522 conn(0x7f11280617a0 msgr2=0x7f1128061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:05.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 1 -- 172.21.15.110:0/4147474522 shutdown_connections 2024-03-25T19:48:05.873 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 1 -- 172.21.15.110:0/4147474522 wait complete. 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 10 monclient: build_initial_monmap 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.871+0000 7f112fc86640 10 monclient: monmap: 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.872+0000 7f112fc86640 1 librados: starting msgr at 2024-03-25T19:48:05.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.872+0000 7f112fc86640 1 librados: starting objecter 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.872+0000 7f112fc86640 1 Processor -- start 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.872+0000 7f112fc86640 1 -- start start 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.872+0000 7f112fc86640 1 librados: setting wanted keys 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.872+0000 7f112fc86640 1 librados: calling monclient init 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.872+0000 7f112fc86640 10 monclient: init 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 10 monclient(hunting): picked mon.noname-c con 0x7f11280b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 10 monclient(hunting): picked mon.noname-a con 0x7f11280b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 10 monclient(hunting): picked mon.noname-b con 0x7f112814be90 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1128142060 con 0x7f11280b4d60 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11281409a0 con 0x7f112814be90 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1128137cd0 con 0x7f11280b1730 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:05.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.873+0000 7f112fc86640 10 monclient(hunting): authenticate will time out at 4657.014049s 2024-03-25T19:48:05.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.881+0000 7f112d221640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f11280b4d60 0x7f11281340d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42600/0 (socket says 172.21.15.110:42600) 2024-03-25T19:48:05.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.881+0000 7f112d221640 1 -- 172.21.15.110:0/2629991997 learned_addr learned my addr 172.21.15.110:0/2629991997 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:05.883 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.883+0000 7f111e7fc640 10 client.?.objecter ms_handle_connect 0x7f11280b4d60 2024-03-25T19:48:05.883 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.883+0000 7f111e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.892+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192886802 0 0) 0x7f1128142060 con 0x7f11280b4d60 2024-03-25T19:48:05.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.892+0000 7f111e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.892+0000 7f111e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.892+0000 7f111e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.892+0000 7f111e7fc640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:48:05.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.892+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110c004eb0 con 0x7f11280b4d60 2024-03-25T19:48:05.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.896+0000 7f111e7fc640 10 client.?.objecter ms_handle_connect 0x7f11280b1730 2024-03-25T19:48:05.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.896+0000 7f111e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.897+0000 7f111e7fc640 10 client.?.objecter ms_handle_connect 0x7f112814be90 2024-03-25T19:48:05.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.897+0000 7f111e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.906 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.906+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 257928907 0 0) 0x7f110c004eb0 con 0x7f11280b4d60 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.906+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1128142060 con 0x7f11280b4d60 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.906+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1118001a10 con 0x7f11280b4d60 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.906+0000 7f111e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.907+0000 7f111e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.907+0000 7f111e7fc640 10 monclient(hunting): dump: 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:05.907 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:05.908 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:05.908 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.907+0000 7f111e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:05.908 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.908+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785009154 0 0) 0x7f11281409a0 con 0x7f112814be90 2024-03-25T19:48:05.908 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.908+0000 7f111e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.908 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.908+0000 7f111e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.908 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.908+0000 7f111e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.908 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.908+0000 7f111e7fc640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:48:05.908 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.908+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110c004eb0 con 0x7f112814be90 2024-03-25T19:48:05.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.915+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540545039 0 0) 0x7f1128137cd0 con 0x7f11280b1730 2024-03-25T19:48:05.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.915+0000 7f111e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.915+0000 7f111e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.915+0000 7f111e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.915+0000 7f111e7fc640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:48:05.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.915+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11281409a0 con 0x7f11280b1730 2024-03-25T19:48:05.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3273561793 0 0) 0x7f1128142060 con 0x7f11280b4d60 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 >> v1:172.21.15.110:6790/0 conn(0x7f11280b1730 legacy=0x7f11281375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 >> v1:172.21.15.178:6789/0 conn(0x7f112814be90 legacy=0x7f11281e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 1 monclient: found mon.a 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1128140e40 con 0x7f11280b4d60 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f111e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f112fc86640 5 monclient: authenticate success, global_id 4566 2024-03-25T19:48:05.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f112fc86640 10 monclient: _renew_subs 2024-03-25T19:48:05.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f112fc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:05.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.921+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11281347e0 con 0x7f11280b4d60 2024-03-25T19:48:05.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.922+0000 7f112fc86640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:05.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.922+0000 7f112fc86640 10 monclient: _renew_subs 2024-03-25T19:48:05.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.922+0000 7f112fc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:05.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.922+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1128134b80 con 0x7f11280b4d60 2024-03-25T19:48:05.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.922+0000 7f112fc86640 1 librados: init done 2024-03-25T19:48:05.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.923+0000 7f112fc86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:05.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.923+0000 7f112fc86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:05.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.923+0000 7f112fc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:05.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:05.923+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1128141050 con 0x7f11280b4d60 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.221+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1118001c90 con 0x7f11280b4d60 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.221+0000 7f111e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.221+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1118003a60 con 0x7f11280b4d60 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.221+0000 7f111e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.221+0000 7f111e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.221+0000 7f111e7fc640 10 monclient: dump: 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:06.221 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:06.222 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:06.222 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:06.222 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:06.222 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:06.222 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:06.222 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:06.222 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:06.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.221+0000 7f111ffff640 4 set_mon_vals no callback set 2024-03-25T19:48:06.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.222+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f1118003c80 con 0x7f11280b4d60 2024-03-25T19:48:06.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.223+0000 7f112da22640 10 monclient: get_auth_request con 0x7f110c06de50 auth_method 0 2024-03-25T19:48:06.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.223+0000 7f111e7fc640 10 client.4566.objecter ms_handle_connect 0x7f110c06de50 2024-03-25T19:48:06.229 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.229+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f11180548f0 con 0x7f11280b4d60 2024-03-25T19:48:06.229 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.229+0000 7f111e7fc640 10 client.4566.objecter ms_dispatch 0x7f1128000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:06.229 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.229+0000 7f111e7fc640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:06.229 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.229+0000 7f111e7fc640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:06.229 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.229+0000 7f111e7fc640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-25T19:48:06.229 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.229+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1118050230 con 0x7f11280b4d60 2024-03-25T19:48:06.229 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.229+0000 7f111e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:06.229 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.229+0000 7f111e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:06.479 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.479+0000 7f112fc86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:06.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.479+0000 7f112fc86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:06.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.479+0000 7f112fc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:06.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.479+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1128141ea0 con 0x7f11280b4d60 2024-03-25T19:48:06.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.505+0000 7f111e7fc640 1 -- 172.21.15.110:0/2629991997 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f1118052330 con 0x7f11280b4d60 2024-03-25T19:48:06.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.505+0000 7f111e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:06.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.505+0000 7f111e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:06.505 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:48:06.506 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.507+0000 7f112fc86640 10 librados: watch_flush enter 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.507+0000 7f112fc86640 10 librados: watch_flush exit 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.507+0000 7f112fc86640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.507+0000 7f112fc86640 10 client.4566.objecter successfully canceled tick 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 >> v1:172.21.15.178:6800/1132739058 conn(0x7f110c06de50 legacy=0x7f110c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 10 monclient: shutdown 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 >> v1:172.21.15.110:6789/0 conn(0x7f11280b4d60 legacy=0x7f11281340d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 shutdown_connections 2024-03-25T19:48:06.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 >> 172.21.15.110:0/2629991997 conn(0x7f11280617a0 msgr2=0x7f11280b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:06.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 shutdown_connections 2024-03-25T19:48:06.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 1 -- 172.21.15.110:0/2629991997 wait complete. 2024-03-25T19:48:06.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:06.508+0000 7f112fc86640 1 librados: shutdown 2024-03-25T19:48:06.520 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:13.326 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.440+0000 7f34995cf640 10 monclient: get_monmap_and_config 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.440+0000 7f34995cf640 10 monclient: build_initial_monmap 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.440+0000 7f34995cf640 10 monclient: monmap: 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:48:13.441280+0000 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:48:13.441280+0000 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:13.440 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:13.441 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:48:13.441 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:48:13.441 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 1 Processor -- start 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 1 -- start start 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 10 monclient: init 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 10 monclient(hunting): picked mon.noname-a con 0x7f3494061830 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 10 monclient(hunting): picked mon.noname-b con 0x7f34940709d0 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 10 monclient(hunting): picked mon.noname-c con 0x7f349413c8c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3494071cd0 con 0x7f3494061830 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3494071fb0 con 0x7f349413c8c0 2024-03-25T19:48:13.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349406ff10 con 0x7f34940709d0 2024-03-25T19:48:13.443 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:13.443 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.441+0000 7f34995cf640 10 monclient(hunting): authenticate will time out at 4664.582790s 2024-03-25T19:48:13.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.446+0000 7f34937fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f349413c8c0 0x7f3494072ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:35742/0 (socket says 172.21.15.110:35742) 2024-03-25T19:48:13.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.446+0000 7f34937fe640 1 -- 172.21.15.110:0/263917653 learned_addr learned my addr 172.21.15.110:0/263917653 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:13.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.461+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898101304 0 0) 0x7f3494071cd0 con 0x7f3494061830 2024-03-25T19:48:13.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.461+0000 7f3491ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.461+0000 7f3491ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.461+0000 7f3491ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.461+0000 7f3491ffb640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:48:13.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.461+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347c003520 con 0x7f3494061830 2024-03-25T19:48:13.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.467+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064184498 0 0) 0x7f3494071fb0 con 0x7f349413c8c0 2024-03-25T19:48:13.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.467+0000 7f3491ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.467+0000 7f3491ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.467+0000 7f3491ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.467+0000 7f3491ffb640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:48:13.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.467+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3494071cd0 con 0x7f349413c8c0 2024-03-25T19:48:13.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538096523 0 0) 0x7f347c003520 con 0x7f3494061830 2024-03-25T19:48:13.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 >> v1:172.21.15.110:6790/0 conn(0x7f34940709d0 legacy=0x7f34940723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 >> v1:172.21.15.178:6789/0 conn(0x7f349413c8c0 legacy=0x7f3494072ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 1 monclient: found mon.noname-a 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349414e950 con 0x7f3494061830 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3484001690 con 0x7f3494061830 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f34995cf640 5 monclient: authenticate success, global_id 4569 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f34995cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f3491ffb640 10 monclient: dump: 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:13.469 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:13.470 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:13.470 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.470 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:13.470 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:13.470 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:13.470 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:13.470 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:13.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.468+0000 7f34995cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:13.483 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.483+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34840017f0 con 0x7f3494061830 2024-03-25T19:48:13.483 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.483+0000 7f3491ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.483+0000 7f3491ffb640 1 -- 172.21.15.110:0/263917653 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3484001c00 con 0x7f3494061830 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.483+0000 7f34995cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.483+0000 7f34995cf640 4 set_mon_vals no callback set 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.484+0000 7f3491ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.484+0000 7f3491ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.484+0000 7f3491ffb640 10 monclient: dump: 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:13.484 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:13.485 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.484+0000 7f34995cf640 10 monclient: shutdown 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.484+0000 7f34995cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.484+0000 7f34995cf640 1 -- 172.21.15.110:0/263917653 >> v1:172.21.15.110:6789/0 conn(0x7f3494061830 legacy=0x7f349413c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.484+0000 7f34995cf640 1 -- 172.21.15.110:0/263917653 shutdown_connections 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.485+0000 7f34995cf640 1 -- 172.21.15.110:0/263917653 >> 172.21.15.110:0/263917653 conn(0x7f34940b11e0 msgr2=0x7f34940b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.485+0000 7f34995cf640 1 -- 172.21.15.110:0/263917653 shutdown_connections 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.485+0000 7f34995cf640 1 -- 172.21.15.110:0/263917653 wait complete. 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.485+0000 7f34995cf640 10 monclient: build_initial_monmap 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.485+0000 7f34995cf640 10 monclient: monmap: 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:48:13.487 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 1 librados: starting msgr at 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 1 librados: starting objecter 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 1 Processor -- start 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 1 -- start start 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 1 librados: setting wanted keys 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 1 librados: calling monclient init 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 10 monclient: init 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:13.488 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:13.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 10 monclient(hunting): picked mon.noname-b con 0x7f3494061830 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:13.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 10 monclient(hunting): picked mon.noname-a con 0x7f34940709d0 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:13.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.486+0000 7f34995cf640 10 monclient(hunting): picked mon.noname-c con 0x7f349413c8c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:13.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.487+0000 7f34995cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349413eb00 con 0x7f34940709d0 2024-03-25T19:48:13.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.487+0000 7f34995cf640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34941d88d0 con 0x7f3494061830 2024-03-25T19:48:13.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.487+0000 7f34995cf640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3494140410 con 0x7f349413c8c0 2024-03-25T19:48:13.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.487+0000 7f34995cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:13.489 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.487+0000 7f34995cf640 10 monclient(hunting): authenticate will time out at 4664.627881s 2024-03-25T19:48:13.498 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.498+0000 7f34937fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f349413c8c0 0x7f34941dfe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:55608/0 (socket says 172.21.15.110:55608) 2024-03-25T19:48:13.498 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.498+0000 7f34937fe640 1 -- 172.21.15.110:0/3686824270 learned_addr learned my addr 172.21.15.110:0/3686824270 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:13.500 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.500+0000 7f347b7fe640 10 client.?.objecter ms_handle_connect 0x7f349413c8c0 2024-03-25T19:48:13.500 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.500+0000 7f347b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.500 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.500+0000 7f347b7fe640 10 client.?.objecter ms_handle_connect 0x7f3494061830 2024-03-25T19:48:13.500 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.500+0000 7f347b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.507+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718293339 0 0) 0x7f34941d88d0 con 0x7f3494061830 2024-03-25T19:48:13.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.507+0000 7f347b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.507+0000 7f347b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.507+0000 7f347b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.507+0000 7f347b7fe640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:48:13.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.507+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3480004eb0 con 0x7f3494061830 2024-03-25T19:48:13.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.518+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276783486 0 0) 0x7f3494140410 con 0x7f349413c8c0 2024-03-25T19:48:13.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.518+0000 7f347b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.518+0000 7f347b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.518+0000 7f347b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.518+0000 7f347b7fe640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:48:13.519 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.519+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34941d88d0 con 0x7f349413c8c0 2024-03-25T19:48:13.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.524+0000 7f347b7fe640 10 client.?.objecter ms_handle_connect 0x7f34940709d0 2024-03-25T19:48:13.523 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.524+0000 7f347b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.525+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157179024 0 0) 0x7f3480004eb0 con 0x7f3494061830 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.525+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3494140410 con 0x7f3494061830 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.525+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3484001b00 con 0x7f3494061830 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.525+0000 7f347b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.525+0000 7f347b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.525+0000 7f347b7fe640 10 monclient(hunting): dump: 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:13.525 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.526 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:13.526 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:13.526 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:13.526 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:13.526 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:13.526 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.525+0000 7f347b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:13.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2986656069 0 0) 0x7f3494140410 con 0x7f3494061830 2024-03-25T19:48:13.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 >> v1:172.21.15.110:6790/0 conn(0x7f349413c8c0 legacy=0x7f34941dfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 >> v1:172.21.15.110:6789/0 conn(0x7f34940709d0 legacy=0x7f349413d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:13.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 1 monclient: found mon.b 2024-03-25T19:48:13.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:13.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:48:13.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3494143200 con 0x7f3494061830 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f347b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 10 monclient: _renew_subs 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34941d94c0 con 0x7f3494061830 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 10 monclient: _renew_subs 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f349413d950 con 0x7f3494061830 2024-03-25T19:48:13.534 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.533+0000 7f34995cf640 1 librados: init done 2024-03-25T19:48:13.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.534+0000 7f34995cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.534+0000 7f34995cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.534+0000 7f34995cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:48:13.535 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.534+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34940627b0 con 0x7f3494061830 2024-03-25T19:48:13.815 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.814+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34840038a0 con 0x7f3494061830 2024-03-25T19:48:13.815 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.814+0000 7f347b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:13.815 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.814+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f3484003bb0 con 0x7f3494061830 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.814+0000 7f347b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.814+0000 7f347b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.814+0000 7f347b7fe640 10 monclient: dump: 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:13.816 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:13.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.814+0000 7f34917fa640 4 set_mon_vals no callback set 2024-03-25T19:48:13.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.815+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f3484019b80 con 0x7f3494061830 2024-03-25T19:48:13.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.816+0000 7f34927fc640 10 monclient: get_auth_request con 0x7f348006dda0 auth_method 0 2024-03-25T19:48:13.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.816+0000 7f347b7fe640 10 client.4585.objecter ms_handle_connect 0x7f348006dda0 2024-03-25T19:48:13.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.820+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f34840536e0 con 0x7f3494061830 2024-03-25T19:48:13.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.820+0000 7f347b7fe640 10 client.4585.objecter ms_dispatch 0x7f3494000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:13.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.820+0000 7f347b7fe640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:13.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.820+0000 7f347b7fe640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:13.820 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.820+0000 7f347b7fe640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T19:48:13.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.821+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f348404f1f0 con 0x7f3494061830 2024-03-25T19:48:13.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.821+0000 7f347b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.821 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:13.821+0000 7f347b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:14.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.083+0000 7f34995cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:14.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.083+0000 7f34995cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:14.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.083+0000 7f34995cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:48:14.083 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.083+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f349414b700 con 0x7f3494061830 2024-03-25T19:48:14.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.153+0000 7f347b7fe640 1 -- 172.21.15.110:0/3686824270 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f3484051120 con 0x7f3494061830 2024-03-25T19:48:14.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.153+0000 7f347b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:14.153 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.153+0000 7f347b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:14.153 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:48:14.154 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:14.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 10 librados: watch_flush enter 2024-03-25T19:48:14.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 10 librados: watch_flush exit 2024-03-25T19:48:14.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T19:48:14.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 10 client.4585.objecter successfully canceled tick 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 >> v1:172.21.15.178:6800/1132739058 conn(0x7f348006dda0 legacy=0x7f3480070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 10 monclient: shutdown 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 >> v1:172.21.15.178:6789/0 conn(0x7f3494061830 legacy=0x7f34941d8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 shutdown_connections 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 >> 172.21.15.110:0/3686824270 conn(0x7f34940b11e0 msgr2=0x7f3494139410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 shutdown_connections 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 1 -- 172.21.15.110:0/3686824270 wait complete. 2024-03-25T19:48:14.157 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:14.156+0000 7f34995cf640 1 librados: shutdown 2024-03-25T19:48:14.165 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:20.972 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:21.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.087+0000 7f525aa93640 10 monclient: get_monmap_and_config 2024-03-25T19:48:21.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.087+0000 7f525aa93640 10 monclient: build_initial_monmap 2024-03-25T19:48:21.087 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.087+0000 7f525aa93640 10 monclient: monmap: 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:48:21.088780+0000 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:48:21.088780+0000 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:48:21.088 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.088+0000 7f525aa93640 1 Processor -- start 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.088+0000 7f525aa93640 1 -- start start 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.088+0000 7f525aa93640 10 monclient: init 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 10 monclient(hunting): picked mon.noname-a con 0x7f52540b10d0 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 10 monclient(hunting): picked mon.noname-c con 0x7f52540b4360 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 10 monclient(hunting): picked mon.noname-b con 0x7f52540a9470 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540a9ab0 con 0x7f52540b10d0 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540b2400 con 0x7f52540b4360 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540aa2a0 con 0x7f52540a9470 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:21.090 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.089+0000 7f525aa93640 10 monclient(hunting): authenticate will time out at 4672.230358s 2024-03-25T19:48:21.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.091+0000 7f524bfff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f52540b4360 0x7f52540b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:40610/0 (socket says 172.21.15.110:40610) 2024-03-25T19:48:21.091 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.091+0000 7f524bfff640 1 -- 172.21.15.110:0/3204177899 learned_addr learned my addr 172.21.15.110:0/3204177899 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:21.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.104+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482565085 0 0) 0x7f52540b2400 con 0x7f52540b4360 2024-03-25T19:48:21.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.105+0000 7f524b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:21.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.105+0000 7f524b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:21.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.105+0000 7f524b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:21.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.105+0000 7f524b7fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:48:21.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.105+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523c003520 con 0x7f52540b4360 2024-03-25T19:48:21.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 18112617 0 0) 0x7f523c003520 con 0x7f52540b4360 2024-03-25T19:48:21.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 >> v1:172.21.15.110:6790/0 conn(0x7f52540a9470 legacy=0x7f525414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 >> v1:172.21.15.110:6789/0 conn(0x7f52540b10d0 legacy=0x7f52540b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.124 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52540b14b0 con 0x7f52540b4360 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5244001690 con 0x7f52540b4360 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f525aa93640 5 monclient: authenticate success, global_id 4588 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f525aa93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f524b7fe640 10 monclient: dump: 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:21.125 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:21.126 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:21.126 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.126 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:21.126 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:21.126 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:21.126 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:21.126 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:21.126 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.124+0000 7f525aa93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:21.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.131+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52440017f0 con 0x7f52540b4360 2024-03-25T19:48:21.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.131+0000 7f524b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:21.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.131+0000 7f524b7fe640 1 -- 172.21.15.110:0/3204177899 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5244001c00 con 0x7f52540b4360 2024-03-25T19:48:21.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.131+0000 7f525aa93640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:21.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.131+0000 7f525aa93640 4 set_mon_vals no callback set 2024-03-25T19:48:21.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.131+0000 7f524b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.131+0000 7f524b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:21.131 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.131+0000 7f524b7fe640 10 monclient: dump: 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:21.132 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:21.133 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.132+0000 7f525aa93640 10 monclient: shutdown 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.132+0000 7f525aa93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.132+0000 7f525aa93640 1 -- 172.21.15.110:0/3204177899 >> v1:172.21.15.178:6789/0 conn(0x7f52540b4360 legacy=0x7f52540b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.132+0000 7f525aa93640 1 -- 172.21.15.110:0/3204177899 shutdown_connections 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.132+0000 7f525aa93640 1 -- 172.21.15.110:0/3204177899 >> 172.21.15.110:0/3204177899 conn(0x7f52540617a0 msgr2=0x7f5254061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.132+0000 7f525aa93640 1 -- 172.21.15.110:0/3204177899 shutdown_connections 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.132+0000 7f525aa93640 1 -- 172.21.15.110:0/3204177899 wait complete. 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.133+0000 7f525aa93640 10 monclient: build_initial_monmap 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.133+0000 7f525aa93640 10 monclient: monmap: 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:21.134 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.133+0000 7f525aa93640 1 librados: starting msgr at 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.133+0000 7f525aa93640 1 librados: starting objecter 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 1 Processor -- start 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 1 -- start start 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 1 librados: setting wanted keys 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 1 librados: calling monclient init 2024-03-25T19:48:21.135 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 10 monclient: init 2024-03-25T19:48:21.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:21.136 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:21.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 10 monclient(hunting): picked mon.noname-a con 0x7f52540a9470 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:21.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.134+0000 7f525aa93640 10 monclient(hunting): picked mon.noname-c con 0x7f52540b10d0 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:21.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.135+0000 7f525aa93640 10 monclient(hunting): picked mon.noname-b con 0x7f52540b4360 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:21.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.135+0000 7f525aa93640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540aca00 con 0x7f52540a9470 2024-03-25T19:48:21.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.135+0000 7f525aa93640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52541d4cb0 con 0x7f52540b4360 2024-03-25T19:48:21.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.135+0000 7f525aa93640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540ab7c0 con 0x7f52540b10d0 2024-03-25T19:48:21.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.135+0000 7f525aa93640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:21.137 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.135+0000 7f525aa93640 10 monclient(hunting): authenticate will time out at 4672.276003s 2024-03-25T19:48:21.147 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.147+0000 7f524bfff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f52540b10d0 0x7f5254138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:60548/0 (socket says 172.21.15.110:60548) 2024-03-25T19:48:21.147 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.147+0000 7f524bfff640 1 -- 172.21.15.110:0/3927869699 learned_addr learned my addr 172.21.15.110:0/3927869699 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:21.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.149+0000 7f52497fa640 10 client.?.objecter ms_handle_connect 0x7f52540b4360 2024-03-25T19:48:21.148 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.149+0000 7f52497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:21.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.150+0000 7f52497fa640 10 client.?.objecter ms_handle_connect 0x7f52540b10d0 2024-03-25T19:48:21.149 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.150+0000 7f52497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:21.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.156+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393440992 0 0) 0x7f52540ab7c0 con 0x7f52540b10d0 2024-03-25T19:48:21.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.156+0000 7f52497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:21.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.156+0000 7f52497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:21.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.156+0000 7f52497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:21.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.156+0000 7f52497fa640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:48:21.156 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.156+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5240004eb0 con 0x7f52540b10d0 2024-03-25T19:48:21.160 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.160+0000 7f52497fa640 10 client.?.objecter ms_handle_connect 0x7f52540a9470 2024-03-25T19:48:21.160 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.160+0000 7f52497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:21.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.167+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133605589 0 0) 0x7f52541d4cb0 con 0x7f52540b4360 2024-03-25T19:48:21.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.167+0000 7f52497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:21.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.167+0000 7f52497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:21.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.167+0000 7f52497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:21.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.167+0000 7f52497fa640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:48:21.167 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.167+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52540ab7c0 con 0x7f52540b4360 2024-03-25T19:48:21.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.171+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2948835958 0 0) 0x7f5240004eb0 con 0x7f52540b10d0 2024-03-25T19:48:21.171 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52541d4cb0 con 0x7f52540b10d0 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5244001b40 con 0x7f52540b10d0 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 10 monclient(hunting): dump: 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:21.172 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:21.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239087275 0 0) 0x7f52540aca00 con 0x7f52540a9470 2024-03-25T19:48:21.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:21.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:21.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:21.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:48:21.173 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.172+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5240004eb0 con 0x7f52540a9470 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.184+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2131463605 0 0) 0x7f52541d4cb0 con 0x7f52540b10d0 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.184+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 >> v1:172.21.15.178:6789/0 conn(0x7f52540b4360 legacy=0x7f52541d7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 >> v1:172.21.15.110:6789/0 conn(0x7f52540a9470 legacy=0x7f52541d43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 10 monclient: _finish_hunting 0 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 1 monclient: found mon.c 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52540ab950 con 0x7f52540b10d0 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 10 monclient: _finish_auth 0 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 10 monclient: _check_auth_tickets 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f52497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 10 monclient: _renew_subs 2024-03-25T19:48:21.185 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:21.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52540aff60 con 0x7f52540b10d0 2024-03-25T19:48:21.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:21.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 10 monclient: _renew_subs 2024-03-25T19:48:21.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:21.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5254139750 con 0x7f52540b10d0 2024-03-25T19:48:21.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 1 librados: init done 2024-03-25T19:48:21.186 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.185+0000 7f525aa93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:21.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.186+0000 7f525aa93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:21.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.186+0000 7f525aa93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:21.187 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.186+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52541d4e40 con 0x7f52540b10d0 2024-03-25T19:48:21.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.484+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5244004a90 con 0x7f52540b10d0 2024-03-25T19:48:21.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.484+0000 7f52497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:21.484 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.484+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5244001d80 con 0x7f52540b10d0 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.484+0000 7f52497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.484+0000 7f52497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.484+0000 7f52497fa640 10 monclient: dump: 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:21.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.484+0000 7f524affd640 4 set_mon_vals no callback set 2024-03-25T19:48:21.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.485+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f524401ac10 con 0x7f52540b10d0 2024-03-25T19:48:21.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.486+0000 7f525882f640 10 monclient: get_auth_request con 0x7f524006de00 auth_method 0 2024-03-25T19:48:21.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.486+0000 7f52497fa640 10 client.4589.objecter ms_handle_connect 0x7f524006de00 2024-03-25T19:48:21.491 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.491+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f52440547f0 con 0x7f52540b10d0 2024-03-25T19:48:21.491 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.491+0000 7f52497fa640 10 client.4589.objecter ms_dispatch 0x7f5254000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:21.491 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.491+0000 7f52497fa640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:21.491 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.491+0000 7f52497fa640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:21.491 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.491+0000 7f52497fa640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T19:48:21.491 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.491+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5244050130 con 0x7f52540b10d0 2024-03-25T19:48:21.491 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.491+0000 7f52497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:21.491 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.491+0000 7f52497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:21.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.742+0000 7f525aa93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.742+0000 7f525aa93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.742+0000 7f525aa93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:21.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.742+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52540627b0 con 0x7f52540b10d0 2024-03-25T19:48:21.863 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.863+0000 7f52497fa640 1 -- 172.21.15.110:0/3927869699 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f5244052230 con 0x7f52540b10d0 2024-03-25T19:48:21.863 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.863+0000 7f52497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.863 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.863+0000 7f52497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:21.863 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:48:21.863 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.865+0000 7f525aa93640 10 librados: watch_flush enter 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.865+0000 7f525aa93640 10 librados: watch_flush exit 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.865+0000 7f525aa93640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.865+0000 7f525aa93640 10 client.4589.objecter successfully canceled tick 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 >> v1:172.21.15.178:6800/1132739058 conn(0x7f524006de00 legacy=0x7f52400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 10 monclient: shutdown 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 >> v1:172.21.15.110:6790/0 conn(0x7f52540b10d0 legacy=0x7f5254138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 shutdown_connections 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 >> 172.21.15.110:0/3927869699 conn(0x7f52540617a0 msgr2=0x7f52540b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 shutdown_connections 2024-03-25T19:48:21.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 1 -- 172.21.15.110:0/3927869699 wait complete. 2024-03-25T19:48:21.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:21.866+0000 7f525aa93640 1 librados: shutdown 2024-03-25T19:48:21.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:23.639 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:30.923 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:23.689 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:30.972 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:23.722 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:31.006 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:23.735 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:30.990 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:23.742 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:30.997 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:23.763 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:31.047 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:23.769 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:31.024 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:23.771 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:31.056 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:23.801 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:31.085 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.683 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:28.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.795+0000 7f051e3a8640 10 monclient: get_monmap_and_config 2024-03-25T19:48:28.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.795+0000 7f051e3a8640 10 monclient: build_initial_monmap 2024-03-25T19:48:28.795 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.795+0000 7f051e3a8640 10 monclient: monmap: 2024-03-25T19:48:28.795 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:28.795 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:28.795 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:48:28.796513+0000 2024-03-25T19:48:28.796 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:48:28.796513+0000 2024-03-25T19:48:28.796 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:28.796 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:28.796 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:28.796 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:48:28.796 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:48:28.796 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:28.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.796+0000 7f051e3a8640 1 Processor -- start 2024-03-25T19:48:28.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.796+0000 7f051e3a8640 1 -- start start 2024-03-25T19:48:28.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.796+0000 7f051e3a8640 10 monclient: init 2024-03-25T19:48:28.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.796+0000 7f051e3a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:28.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:28.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f05180a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:28.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f051813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:28.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f05181430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:28.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0518142520 con 0x7f05181430c0 2024-03-25T19:48:28.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05180b28a0 con 0x7f05180a98f0 2024-03-25T19:48:28.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05180b2a30 con 0x7f051813f710 2024-03-25T19:48:28.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:28.798 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.797+0000 7f051e3a8640 10 monclient(hunting): authenticate will time out at 4679.938041s 2024-03-25T19:48:28.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.799+0000 7f0517fff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f05180a98f0 0x7f05180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:51474/0 (socket says 172.21.15.110:51474) 2024-03-25T19:48:28.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.799+0000 7f0517fff640 1 -- 172.21.15.110:0/3407585644 learned_addr learned my addr 172.21.15.110:0/3407585644 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:28.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.841+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156812791 0 0) 0x7f0518142520 con 0x7f05181430c0 2024-03-25T19:48:28.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.841+0000 7f0516ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.841+0000 7f0516ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.841+0000 7f0516ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.841+0000 7f0516ffd640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:48:28.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.841+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0500003520 con 0x7f05181430c0 2024-03-25T19:48:28.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2689058166 0 0) 0x7f0500003520 con 0x7f05181430c0 2024-03-25T19:48:28.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 >> v1:172.21.15.110:6790/0 conn(0x7f051813f710 legacy=0x7f05181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 >> v1:172.21.15.178:6789/0 conn(0x7f05180a98f0 legacy=0x7f05180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:28.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 1 monclient: found mon.noname-a 2024-03-25T19:48:28.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:28.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05180b44c0 con 0x7f05181430c0 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 10 monclient: _finish_auth 0 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0510001a10 con 0x7f05181430c0 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f051e3a8640 5 monclient: authenticate success, global_id 4581 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f051e3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f0516ffd640 10 monclient: dump: 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.897 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.898 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:28.898 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:28.898 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:28.898 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:28.898 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:28.898 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.896+0000 7f051e3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:28.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.924+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0510001b70 con 0x7f05181430c0 2024-03-25T19:48:28.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.924+0000 7f0516ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:28.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.924+0000 7f0516ffd640 1 -- 172.21.15.110:0/3407585644 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0510001f80 con 0x7f05181430c0 2024-03-25T19:48:28.924 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.924+0000 7f051e3a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.924+0000 7f051e3a8640 4 set_mon_vals no callback set 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.924+0000 7f0516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.924+0000 7f0516ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.924+0000 7f0516ffd640 10 monclient: dump: 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:28.925 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.925+0000 7f051e3a8640 10 monclient: shutdown 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.925+0000 7f051e3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.925+0000 7f051e3a8640 1 -- 172.21.15.110:0/3407585644 >> v1:172.21.15.110:6789/0 conn(0x7f05181430c0 legacy=0x7f051814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.925+0000 7f051e3a8640 1 -- 172.21.15.110:0/3407585644 shutdown_connections 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.925+0000 7f051e3a8640 1 -- 172.21.15.110:0/3407585644 >> 172.21.15.110:0/3407585644 conn(0x7f05180b7ea0 msgr2=0x7f05180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.925+0000 7f051e3a8640 1 -- 172.21.15.110:0/3407585644 shutdown_connections 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.925+0000 7f051e3a8640 1 -- 172.21.15.110:0/3407585644 wait complete. 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.926+0000 7f051e3a8640 10 monclient: build_initial_monmap 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.926+0000 7f051e3a8640 10 monclient: monmap: 2024-03-25T19:48:28.927 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.927+0000 7f051e3a8640 1 librados: starting msgr at 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.927+0000 7f051e3a8640 1 librados: starting objecter 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.927+0000 7f051e3a8640 1 Processor -- start 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.927+0000 7f051e3a8640 1 -- start start 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.927+0000 7f051e3a8640 1 librados: setting wanted keys 2024-03-25T19:48:28.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.927+0000 7f051e3a8640 1 librados: calling monclient init 2024-03-25T19:48:28.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.927+0000 7f051e3a8640 10 monclient: init 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f05180a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f051813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f05181430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0518136ff0 con 0x7f05181430c0 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05180b7860 con 0x7f05180a98f0 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0518135070 con 0x7f051813f710 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:28.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.928+0000 7f051e3a8640 10 monclient(hunting): authenticate will time out at 4680.069624s 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.944+0000 7f051c945640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f05181430c0 0x7f05181db210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:57864/0 (socket says 172.21.15.110:57864) 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.944+0000 7f051c945640 1 -- 172.21.15.110:0/327604082 learned_addr learned my addr 172.21.15.110:0/327604082 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:28.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.947+0000 7f0514ff9640 10 client.?.objecter ms_handle_connect 0x7f05181430c0 2024-03-25T19:48:28.947 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.947+0000 7f0514ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.952+0000 7f0514ff9640 10 client.?.objecter ms_handle_connect 0x7f05180a98f0 2024-03-25T19:48:28.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.952+0000 7f0514ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.961+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607879754 0 0) 0x7f0518136ff0 con 0x7f05181430c0 2024-03-25T19:48:28.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.961+0000 7f0514ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.961+0000 7f0514ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.961+0000 7f0514ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.961+0000 7f0514ff9640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:48:28.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.961+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504004eb0 con 0x7f05181430c0 2024-03-25T19:48:28.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.970+0000 7f0514ff9640 10 client.?.objecter ms_handle_connect 0x7f051813f710 2024-03-25T19:48:28.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.970+0000 7f0514ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.974+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517502522 0 0) 0x7f05180b7860 con 0x7f05180a98f0 2024-03-25T19:48:28.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.974+0000 7f0514ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.974+0000 7f0514ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.974+0000 7f0514ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.974+0000 7f0514ff9640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:48:28.974 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.974+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0518136ff0 con 0x7f05180a98f0 2024-03-25T19:48:28.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.980+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435013226 0 0) 0x7f0518135070 con 0x7f051813f710 2024-03-25T19:48:28.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.980+0000 7f0514ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.980+0000 7f0514ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.980+0000 7f0514ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.980+0000 7f0514ff9640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:48:28.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.980+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05180b7860 con 0x7f051813f710 2024-03-25T19:48:28.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.990+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518182828 0 0) 0x7f0518136ff0 con 0x7f05180a98f0 2024-03-25T19:48:28.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.990+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0518135070 con 0x7f05180a98f0 2024-03-25T19:48:28.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.990+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f05080018a0 con 0x7f05180a98f0 2024-03-25T19:48:28.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.990+0000 7f0514ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.990+0000 7f0514ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:28.990 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.990+0000 7f0514ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:28.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.990+0000 7f0514ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:28.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.993+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1098521855 0 0) 0x7f05180b7860 con 0x7f051813f710 2024-03-25T19:48:28.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.993+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0518136ff0 con 0x7f051813f710 2024-03-25T19:48:28.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f050c0018e0 con 0x7f051813f710 2024-03-25T19:48:28.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:28.995 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:28.995 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:28.995 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:28.995 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.995 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3368351232 0 0) 0x7f0504004eb0 con 0x7f05181430c0 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05180b7860 con 0x7f05181430c0 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f0510001fc0 con 0x7f05181430c0 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:28.994+0000 7f0514ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:28.996 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:28.997 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.997 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:28.997 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.997 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:28.997 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:28.997 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:28.997 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:28.997 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:29.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 626386762 0 0) 0x7f0518136ff0 con 0x7f051813f710 2024-03-25T19:48:29.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 >> v1:172.21.15.178:6789/0 conn(0x7f05180a98f0 legacy=0x7f0518130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:29.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 >> v1:172.21.15.110:6789/0 conn(0x7f05181430c0 legacy=0x7f05181db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:29.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:48:29.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 1 monclient: found mon.c 2024-03-25T19:48:29.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:29.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0518135200 con 0x7f051813f710 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 10 monclient: _finish_auth 0 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f0514ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 5 monclient: authenticate success, global_id 4595 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 10 monclient: _renew_subs 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0518135550 con 0x7f051813f710 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 10 monclient: _renew_subs 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0518131840 con 0x7f051813f710 2024-03-25T19:48:29.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.004+0000 7f051e3a8640 1 librados: init done 2024-03-25T19:48:29.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.005+0000 7f051e3a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:29.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.005+0000 7f051e3a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:29.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.005+0000 7f051e3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:29.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.005+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0518135390 con 0x7f051813f710 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.323+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f050c0035f0 con 0x7f051813f710 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.323+0000 7f0514ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.323+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f050c003760 con 0x7f051813f710 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.323+0000 7f0514ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.323+0000 7f0514ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.323+0000 7f0514ff9640 10 monclient: dump: 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:29.323 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:29.324 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:29.324 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:29.324 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:29.324 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:29.324 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:29.324 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:29.324 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:29.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.323+0000 7f05167fc640 4 set_mon_vals no callback set 2024-03-25T19:48:29.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.324+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f050c019730 con 0x7f051813f710 2024-03-25T19:48:29.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.324+0000 7f0517fff640 10 monclient: get_auth_request con 0x7f050406e110 auth_method 0 2024-03-25T19:48:29.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.325+0000 7f0514ff9640 10 client.4595.objecter ms_handle_connect 0x7f050406e110 2024-03-25T19:48:29.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.330+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f050c053370 con 0x7f051813f710 2024-03-25T19:48:29.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.330+0000 7f0514ff9640 10 client.4595.objecter ms_dispatch 0x7f0518000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:29.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.330+0000 7f0514ff9640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:29.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.330+0000 7f0514ff9640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:29.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.330+0000 7f0514ff9640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-25T19:48:29.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.330+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f050c04ee80 con 0x7f051813f710 2024-03-25T19:48:29.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.330+0000 7f0514ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:29.331 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.330+0000 7f0514ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:29.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.578+0000 7f051e3a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:29.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.578+0000 7f051e3a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:29.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.578+0000 7f051e3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:29.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.578+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05180b79f0 con 0x7f051813f710 2024-03-25T19:48:29.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.621+0000 7f0514ff9640 1 -- 172.21.15.110:0/327604082 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f050c050db0 con 0x7f051813f710 2024-03-25T19:48:29.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.621+0000 7f0514ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:29.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.621+0000 7f0514ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:29.622 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:48:29.622 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:29.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 10 librados: watch_flush enter 2024-03-25T19:48:29.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 10 librados: watch_flush exit 2024-03-25T19:48:29.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-25T19:48:29.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 10 client.4595.objecter successfully canceled tick 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 >> v1:172.21.15.178:6800/1132739058 conn(0x7f050406e110 legacy=0x7f05040705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 10 monclient: shutdown 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 >> v1:172.21.15.110:6790/0 conn(0x7f051813f710 legacy=0x7f05181dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 shutdown_connections 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 >> 172.21.15.110:0/327604082 conn(0x7f05180b7ea0 msgr2=0x7f0518074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 shutdown_connections 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 1 -- 172.21.15.110:0/327604082 wait complete. 2024-03-25T19:48:29.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:29.624+0000 7f051e3a8640 1 librados: shutdown 2024-03-25T19:48:29.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:36.443 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:36.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.560+0000 7f9920645640 10 monclient: get_monmap_and_config 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.560+0000 7f9920645640 10 monclient: build_initial_monmap 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.560+0000 7f9920645640 10 monclient: monmap: 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:48:36.560975+0000 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:48:36.560975+0000 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:48:36.560 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.560+0000 7f9920645640 1 Processor -- start 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.560+0000 7f9920645640 1 -- start start 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.560+0000 7f9920645640 10 monclient: init 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.561+0000 7f9920645640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.561+0000 7f9920645640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.561+0000 7f9920645640 10 monclient(hunting): picked mon.noname-a con 0x7f99180a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.562+0000 7f9920645640 10 monclient(hunting): picked mon.noname-b con 0x7f991813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.562+0000 7f9920645640 10 monclient(hunting): picked mon.noname-c con 0x7f99181430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:36.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.562+0000 7f9920645640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9918142520 con 0x7f99180a98f0 2024-03-25T19:48:36.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.562+0000 7f9920645640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99180b28a0 con 0x7f99181430c0 2024-03-25T19:48:36.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.562+0000 7f9920645640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99180b2a30 con 0x7f991813f710 2024-03-25T19:48:36.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.562+0000 7f9920645640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:36.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.562+0000 7f9920645640 10 monclient(hunting): authenticate will time out at 4687.703142s 2024-03-25T19:48:36.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.567+0000 7f991dbe0640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f991813f710 0x7f99181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:42998/0 (socket says 172.21.15.110:42998) 2024-03-25T19:48:36.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.567+0000 7f991dbe0640 1 -- 172.21.15.110:0/276465582 learned_addr learned my addr 172.21.15.110:0/276465582 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:36.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.580+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662458257 0 0) 0x7f99180b2a30 con 0x7f991813f710 2024-03-25T19:48:36.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.580+0000 7f991d3df640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.580+0000 7f991d3df640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.580+0000 7f991d3df640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.580+0000 7f991d3df640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:48:36.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.580+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9904003520 con 0x7f991813f710 2024-03-25T19:48:36.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.589+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297779606 0 0) 0x7f99180b28a0 con 0x7f99181430c0 2024-03-25T19:48:36.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.589+0000 7f991d3df640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.589+0000 7f991d3df640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.589+0000 7f991d3df640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.589+0000 7f991d3df640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:48:36.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.589+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99180b2a30 con 0x7f99181430c0 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390790134 0 0) 0x7f9904003520 con 0x7f991813f710 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 >> v1:172.21.15.178:6789/0 conn(0x7f99181430c0 legacy=0x7f991814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 >> v1:172.21.15.110:6789/0 conn(0x7f99180a98f0 legacy=0x7f99180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 10 monclient: _finish_hunting 0 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 1 monclient: found mon.noname-b 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99180b44c0 con 0x7f991813f710 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 10 monclient: _finish_auth 0 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 10 monclient: _check_auth_tickets 2024-03-25T19:48:36.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9908001690 con 0x7f991813f710 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f991d3df640 10 monclient: dump: 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:36.598 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:36.599 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:36.599 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f9920645640 5 monclient: authenticate success, global_id 4598 2024-03-25T19:48:36.599 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.597+0000 7f9920645640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:36.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.602+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99080017f0 con 0x7f991813f710 2024-03-25T19:48:36.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.602+0000 7f991d3df640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:36.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.602+0000 7f991d3df640 1 -- 172.21.15.110:0/276465582 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9908001c00 con 0x7f991813f710 2024-03-25T19:48:36.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.602+0000 7f991d3df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.602+0000 7f991d3df640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:36.602 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.602+0000 7f991d3df640 10 monclient: dump: 2024-03-25T19:48:36.602 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:36.602 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.602+0000 7f9920645640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:36.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.602+0000 7f9920645640 4 set_mon_vals no callback set 2024-03-25T19:48:36.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.603+0000 7f9920645640 10 monclient: shutdown 2024-03-25T19:48:36.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.603+0000 7f9920645640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:36.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.603+0000 7f9920645640 1 -- 172.21.15.110:0/276465582 >> v1:172.21.15.110:6790/0 conn(0x7f991813f710 legacy=0x7f99181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.603+0000 7f9920645640 1 -- 172.21.15.110:0/276465582 shutdown_connections 2024-03-25T19:48:36.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.603+0000 7f9920645640 1 -- 172.21.15.110:0/276465582 >> 172.21.15.110:0/276465582 conn(0x7f99180b7ea0 msgr2=0x7f99180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:36.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.603+0000 7f9920645640 1 -- 172.21.15.110:0/276465582 shutdown_connections 2024-03-25T19:48:36.605 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.603+0000 7f9920645640 1 -- 172.21.15.110:0/276465582 wait complete. 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.604+0000 7f9920645640 10 monclient: build_initial_monmap 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.604+0000 7f9920645640 10 monclient: monmap: 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.604+0000 7f9920645640 1 librados: starting msgr at 2024-03-25T19:48:36.606 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.604+0000 7f9920645640 1 librados: starting objecter 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.604+0000 7f9920645640 1 Processor -- start 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 1 -- start start 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 1 librados: setting wanted keys 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 1 librados: calling monclient init 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 10 monclient: init 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 10 monclient(hunting): picked mon.noname-a con 0x7f99180a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 10 monclient(hunting): picked mon.noname-c con 0x7f991813f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 10 monclient(hunting): picked mon.noname-b con 0x7f99181430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991806fe20 con 0x7f99180a98f0 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9918070100 con 0x7f99181430c0 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991806e820 con 0x7f991813f710 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:36.607 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.605+0000 7f9920645640 10 monclient(hunting): authenticate will time out at 4687.746605s 2024-03-25T19:48:36.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.609+0000 7f991dbe0640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f991813f710 0x7f991814c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:43042/0 (socket says 172.21.15.110:43042) 2024-03-25T19:48:36.608 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.609+0000 7f991dbe0640 1 -- 172.21.15.110:0/1303295279 learned_addr learned my addr 172.21.15.110:0/1303295279 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:36.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.609+0000 7f990effd640 10 client.?.objecter ms_handle_connect 0x7f991813f710 2024-03-25T19:48:36.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.609+0000 7f990effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:36.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.619+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145759838 0 0) 0x7f991806e820 con 0x7f991813f710 2024-03-25T19:48:36.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.619+0000 7f990effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.619+0000 7f990effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.619+0000 7f990effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.619+0000 7f990effd640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:48:36.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.619+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98fc004eb0 con 0x7f991813f710 2024-03-25T19:48:36.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.621+0000 7f990effd640 10 client.?.objecter ms_handle_connect 0x7f99181430c0 2024-03-25T19:48:36.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.621+0000 7f990effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:36.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.635+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597060639 0 0) 0x7f9918070100 con 0x7f99181430c0 2024-03-25T19:48:36.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.635+0000 7f990effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.635+0000 7f990effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.635+0000 7f990effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.635+0000 7f990effd640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:48:36.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.635+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f991806e820 con 0x7f99181430c0 2024-03-25T19:48:36.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.638+0000 7f990effd640 10 client.?.objecter ms_handle_connect 0x7f99180a98f0 2024-03-25T19:48:36.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.638+0000 7f990effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:36.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.648+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694898241 0 0) 0x7f991806fe20 con 0x7f99180a98f0 2024-03-25T19:48:36.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.648+0000 7f990effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.648+0000 7f990effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.648+0000 7f990effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.648+0000 7f990effd640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:48:36.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.648+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9918070100 con 0x7f99180a98f0 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.650+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 485380047 0 0) 0x7f991806e820 con 0x7f99181430c0 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.650+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f991806fe20 con 0x7f99181430c0 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.650+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9910001b50 con 0x7f99181430c0 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.650+0000 7f990effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.650+0000 7f990effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.650+0000 7f990effd640 10 monclient(hunting): dump: 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.650 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:36.651 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:36.651 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:36.651 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:36.651 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:36.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.650+0000 7f990effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.652+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2998456312 0 0) 0x7f98fc004eb0 con 0x7f991813f710 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.652+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f991806e820 con 0x7f991813f710 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.652+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9908001bc0 con 0x7f991813f710 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.652+0000 7f990effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.652+0000 7f990effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.652+0000 7f990effd640 10 monclient(hunting): dump: 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.652 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.653 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.653 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:36.653 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:36.653 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:36.653 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:36.653 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1549226184 0 0) 0x7f991806e820 con 0x7f991813f710 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 >> v1:172.21.15.178:6789/0 conn(0x7f99181430c0 legacy=0x7f991814ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 >> v1:172.21.15.110:6789/0 conn(0x7f99180a98f0 legacy=0x7f991806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 1 monclient: found mon.c 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991806e9b0 con 0x7f991813f710 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 10 monclient: _finish_auth 0 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f990effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:36.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f9920645640 5 monclient: authenticate success, global_id 4601 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f9920645640 10 monclient: _renew_subs 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f9920645640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.669+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99181e7700 con 0x7f991813f710 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 10 monclient: _renew_subs 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99181e7a70 con 0x7f991813f710 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 1 librados: init done 2024-03-25T19:48:36.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:36.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:36.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:36.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.670+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99180627b0 con 0x7f991813f710 2024-03-25T19:48:36.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.960+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99080011f0 con 0x7f991813f710 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.960+0000 7f990effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.961+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9908004c30 con 0x7f991813f710 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.961+0000 7f990effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.961+0000 7f990effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.961+0000 7f990effd640 10 monclient: dump: 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:36.961 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:36.962 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:36.962 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:36.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.961+0000 7f991cbde640 4 set_mon_vals no callback set 2024-03-25T19:48:36.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.962+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f990801ac80 con 0x7f991813f710 2024-03-25T19:48:36.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.962+0000 7f991e3e1640 10 monclient: get_auth_request con 0x7f98fc06df70 auth_method 0 2024-03-25T19:48:36.962 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.963+0000 7f990effd640 10 client.4601.objecter ms_handle_connect 0x7f98fc06df70 2024-03-25T19:48:36.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.967+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f9908054910 con 0x7f991813f710 2024-03-25T19:48:36.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.967+0000 7f990effd640 10 client.4601.objecter ms_dispatch 0x7f9918000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:36.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.967+0000 7f990effd640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:36.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.967+0000 7f990effd640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:36.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.967+0000 7f990effd640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T19:48:36.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.967+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9908050250 con 0x7f991813f710 2024-03-25T19:48:36.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.967+0000 7f990effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:36.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:36.967+0000 7f990effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:37.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.213+0000 7f9920645640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:37.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.213+0000 7f9920645640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:37.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.213+0000 7f9920645640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:37.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.213+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99181434a0 con 0x7f991813f710 2024-03-25T19:48:37.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.241+0000 7f990effd640 1 -- 172.21.15.110:0/1303295279 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f9908052350 con 0x7f991813f710 2024-03-25T19:48:37.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.241+0000 7f990effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:37.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.241+0000 7f990effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:37.241 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:48:37.241 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:37.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 10 librados: watch_flush enter 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 10 librados: watch_flush exit 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 10 client.4601.objecter successfully canceled tick 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 >> v1:172.21.15.178:6800/1132739058 conn(0x7f98fc06df70 legacy=0x7f98fc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 10 monclient: shutdown 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 >> v1:172.21.15.110:6790/0 conn(0x7f991813f710 legacy=0x7f991814c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.243+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 shutdown_connections 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.244+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 >> 172.21.15.110:0/1303295279 conn(0x7f99180b7ea0 msgr2=0x7f9918074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.244+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 shutdown_connections 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.244+0000 7f9920645640 1 -- 172.21.15.110:0/1303295279 wait complete. 2024-03-25T19:48:37.244 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:37.244+0000 7f9920645640 1 librados: shutdown 2024-03-25T19:48:37.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:43.590 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:50.874 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.596 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:50.880 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.634 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:50.918 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.636 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:50.891 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.652 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:50.936 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.652 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:50.907 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.659 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:50.943 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.679 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:08:50.933 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.688 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:50.972 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:44.062 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:44.180 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.181+0000 7f4b6590e640 10 monclient: get_monmap_and_config 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.181+0000 7f4b6590e640 10 monclient: build_initial_monmap 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.181+0000 7f4b6590e640 10 monclient: monmap: 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:48:44.181991+0000 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:48:44.181991+0000 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:48:44.181 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:44.182 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.181+0000 7f4b6590e640 1 Processor -- start 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.181+0000 7f4b6590e640 1 -- start start 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.181+0000 7f4b6590e640 10 monclient: init 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 10 monclient(hunting): picked mon.noname-c con 0x7f4b600b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 10 monclient(hunting): picked mon.noname-b con 0x7f4b600b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b6014be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600af940 con 0x7f4b6014be90 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600b0a70 con 0x7f4b600b4d60 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600b0d50 con 0x7f4b600b1730 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:44.183 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.182+0000 7f4b6590e640 10 monclient(hunting): authenticate will time out at 4695.323301s 2024-03-25T19:48:44.184 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.185+0000 7f4b5e7fc640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f4b600b1730 0x7f4b600b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:35314/0 (socket says 172.21.15.110:35314) 2024-03-25T19:48:44.184 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.185+0000 7f4b5e7fc640 1 -- 172.21.15.110:0/2730822629 learned_addr learned my addr 172.21.15.110:0/2730822629 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:44.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.212+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132491907 0 0) 0x7f4b600b0d50 con 0x7f4b600b1730 2024-03-25T19:48:44.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.212+0000 7f4b5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.211 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.212+0000 7f4b5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.212+0000 7f4b5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.212+0000 7f4b5dffb640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:48:44.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.212+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b48003520 con 0x7f4b600b1730 2024-03-25T19:48:44.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.214+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174633375 0 0) 0x7f4b600b0a70 con 0x7f4b600b4d60 2024-03-25T19:48:44.214 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.214+0000 7f4b5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.214 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.214+0000 7f4b5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.214 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.214+0000 7f4b5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.214 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.214+0000 7f4b5dffb640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:48:44.214 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.214+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b600b0d50 con 0x7f4b600b4d60 2024-03-25T19:48:44.219 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.220+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706827260 0 0) 0x7f4b600af940 con 0x7f4b6014be90 2024-03-25T19:48:44.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.220+0000 7f4b5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.220+0000 7f4b5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.220+0000 7f4b5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.220+0000 7f4b5dffb640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:48:44.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.220+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b600b0a70 con 0x7f4b6014be90 2024-03-25T19:48:44.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 515490572 0 0) 0x7f4b600b0d50 con 0x7f4b600b4d60 2024-03-25T19:48:44.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 >> v1:172.21.15.110:6790/0 conn(0x7f4b600b1730 legacy=0x7f4b600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 >> v1:172.21.15.110:6789/0 conn(0x7f4b6014be90 legacy=0x7f4b6014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:44.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 1 monclient: found mon.noname-c 2024-03-25T19:48:44.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:44.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b600b1030 con 0x7f4b600b4d60 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4b50001670 con 0x7f4b600b4d60 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b5dffb640 10 monclient: dump: 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.222 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:44.223 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:44.223 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:44.223 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:44.223 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:44.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b6590e640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:48:44.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.221+0000 7f4b6590e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:44.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.242+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b500017d0 con 0x7f4b600b4d60 2024-03-25T19:48:44.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.242+0000 7f4b5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:44.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.242+0000 7f4b5dffb640 1 -- 172.21.15.110:0/2730822629 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4b50001be0 con 0x7f4b600b4d60 2024-03-25T19:48:44.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.242+0000 7f4b5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.242+0000 7f4b5dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.242+0000 7f4b5dffb640 10 monclient: dump: 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.242+0000 7f4b6590e640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:44.243 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.242+0000 7f4b6590e640 4 set_mon_vals no callback set 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.243+0000 7f4b6590e640 10 monclient: shutdown 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.243+0000 7f4b6590e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.243+0000 7f4b6590e640 1 -- 172.21.15.110:0/2730822629 >> v1:172.21.15.178:6789/0 conn(0x7f4b600b4d60 legacy=0x7f4b6013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.243+0000 7f4b6590e640 1 -- 172.21.15.110:0/2730822629 shutdown_connections 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.243+0000 7f4b6590e640 1 -- 172.21.15.110:0/2730822629 >> 172.21.15.110:0/2730822629 conn(0x7f4b600617a0 msgr2=0x7f4b60061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.243+0000 7f4b6590e640 1 -- 172.21.15.110:0/2730822629 shutdown_connections 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.243+0000 7f4b6590e640 1 -- 172.21.15.110:0/2730822629 wait complete. 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.244+0000 7f4b6590e640 10 monclient: build_initial_monmap 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.244+0000 7f4b6590e640 10 monclient: monmap: 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:48:44.245 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.245+0000 7f4b6590e640 1 librados: starting msgr at 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.245+0000 7f4b6590e640 1 librados: starting objecter 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.245+0000 7f4b6590e640 1 Processor -- start 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.245+0000 7f4b6590e640 1 -- start start 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.245+0000 7f4b6590e640 1 librados: setting wanted keys 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.245+0000 7f4b6590e640 1 librados: calling monclient init 2024-03-25T19:48:44.246 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.245+0000 7f4b6590e640 10 monclient: init 2024-03-25T19:48:44.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:44.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:44.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 10 monclient(hunting): picked mon.noname-b con 0x7f4b600b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:44.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 10 monclient(hunting): picked mon.noname-c con 0x7f4b600b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:44.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b6014be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:44.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b60140380 con 0x7f4b6014be90 2024-03-25T19:48:44.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b60141a70 con 0x7f4b600b1730 2024-03-25T19:48:44.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6013b160 con 0x7f4b600b4d60 2024-03-25T19:48:44.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:44.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.246+0000 7f4b6590e640 10 monclient(hunting): authenticate will time out at 4695.387060s 2024-03-25T19:48:44.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.252+0000 7f4b5f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4b6014be90 0x7f4b601d7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60932/0 (socket says 172.21.15.110:60932) 2024-03-25T19:48:44.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.252+0000 7f4b5f7fe640 1 -- 172.21.15.110:0/4004108207 learned_addr learned my addr 172.21.15.110:0/4004108207 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:44.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.252+0000 7f4b47fff640 10 client.?.objecter ms_handle_connect 0x7f4b6014be90 2024-03-25T19:48:44.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.252+0000 7f4b47fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:44.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.255+0000 7f4b47fff640 10 client.?.objecter ms_handle_connect 0x7f4b600b1730 2024-03-25T19:48:44.254 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.255+0000 7f4b47fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:44.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.261+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907421334 0 0) 0x7f4b60140380 con 0x7f4b6014be90 2024-03-25T19:48:44.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.261+0000 7f4b47fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.261+0000 7f4b47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.261+0000 7f4b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.261+0000 7f4b47fff640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:48:44.261 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.261+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b4c004eb0 con 0x7f4b6014be90 2024-03-25T19:48:44.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.266+0000 7f4b47fff640 10 client.?.objecter ms_handle_connect 0x7f4b600b4d60 2024-03-25T19:48:44.265 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.266+0000 7f4b47fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:44.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.268+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222144291 0 0) 0x7f4b60141a70 con 0x7f4b600b1730 2024-03-25T19:48:44.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.268+0000 7f4b47fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.268 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.268+0000 7f4b47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.268+0000 7f4b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.268+0000 7f4b47fff640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:48:44.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.268+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b60140380 con 0x7f4b600b1730 2024-03-25T19:48:44.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.272+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3347246283 0 0) 0x7f4b4c004eb0 con 0x7f4b6014be90 2024-03-25T19:48:44.272 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.272+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b60141a70 con 0x7f4b6014be90 2024-03-25T19:48:44.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.273+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4b58001d30 con 0x7f4b6014be90 2024-03-25T19:48:44.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.274+0000 7f4b47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.274+0000 7f4b47fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:44.273 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.274+0000 7f4b47fff640 10 monclient(hunting): dump: 2024-03-25T19:48:44.273 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:44.274 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.274+0000 7f4b47fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:44.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.280+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086886924 0 0) 0x7f4b6013b160 con 0x7f4b600b4d60 2024-03-25T19:48:44.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.280+0000 7f4b47fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.280+0000 7f4b47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.280+0000 7f4b47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.280+0000 7f4b47fff640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:48:44.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.280+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b4c004eb0 con 0x7f4b600b4d60 2024-03-25T19:48:44.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.283+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274746433 0 0) 0x7f4b60140380 con 0x7f4b600b1730 2024-03-25T19:48:44.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.283+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b6013b160 con 0x7f4b600b1730 2024-03-25T19:48:44.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.283+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4b50001ae0 con 0x7f4b600b1730 2024-03-25T19:48:44.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.283+0000 7f4b47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.283+0000 7f4b47fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:44.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.283+0000 7f4b47fff640 10 monclient(hunting): dump: 2024-03-25T19:48:44.283 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:44.283 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:44.284 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.284 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.284 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.284 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:44.284 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:44.284 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:44.284 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:44.284 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:44.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 682916741 0 0) 0x7f4b60141a70 con 0x7f4b6014be90 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 >> v1:172.21.15.110:6790/0 conn(0x7f4b600b4d60 legacy=0x7f4b601d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 >> v1:172.21.15.178:6789/0 conn(0x7f4b600b1730 legacy=0x7f4b60134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 1 monclient: found mon.a 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6013b9d0 con 0x7f4b6014be90 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 10 monclient: _finish_auth 0 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 5 monclient: authenticate success, global_id 4596 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 10 monclient: _renew_subs 2024-03-25T19:48:44.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b60135450 con 0x7f4b6014be90 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 10 monclient: _renew_subs 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b601421e0 con 0x7f4b6014be90 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.284+0000 7f4b6590e640 1 librados: init done 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.286+0000 7f4b6590e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.286+0000 7f4b6590e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.286+0000 7f4b6590e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:44.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.286+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6013bc20 con 0x7f4b6014be90 2024-03-25T19:48:44.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.579+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b58003970 con 0x7f4b6014be90 2024-03-25T19:48:44.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.579+0000 7f4b47fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:44.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.579+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f4b58003dc0 con 0x7f4b6014be90 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.579+0000 7f4b47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.579+0000 7f4b47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.579+0000 7f4b47fff640 10 monclient: dump: 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:44.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.579+0000 7f4b5d7fa640 4 set_mon_vals no callback set 2024-03-25T19:48:44.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.580+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f4b58003fe0 con 0x7f4b6014be90 2024-03-25T19:48:44.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.581+0000 7f4b5effd640 10 monclient: get_auth_request con 0x7f4b4c06dfb0 auth_method 0 2024-03-25T19:48:44.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.581+0000 7f4b47fff640 10 client.4596.objecter ms_handle_connect 0x7f4b4c06dfb0 2024-03-25T19:48:44.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.587+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f4b58054a50 con 0x7f4b6014be90 2024-03-25T19:48:44.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.587+0000 7f4b47fff640 10 client.4596.objecter ms_dispatch 0x7f4b60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:44.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.587+0000 7f4b47fff640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:44.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.587+0000 7f4b47fff640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:44.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.587+0000 7f4b47fff640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-25T19:48:44.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.587+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b58050560 con 0x7f4b6014be90 2024-03-25T19:48:44.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.587+0000 7f4b47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:44.587 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.587+0000 7f4b47fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:44.838 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.838+0000 7f4b6590e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.838+0000 7f4b6590e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.838+0000 7f4b6590e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:44.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.838+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b600627b0 con 0x7f4b6014be90 2024-03-25T19:48:44.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.871+0000 7f4b47fff640 1 -- 172.21.15.110:0/4004108207 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f4b58052490 con 0x7f4b6014be90 2024-03-25T19:48:44.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.871+0000 7f4b47fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.871 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.871+0000 7f4b47fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:44.871 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:48:44.871 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 10 librados: watch_flush enter 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 10 librados: watch_flush exit 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 10 client.4596.objecter successfully canceled tick 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 >> v1:172.21.15.178:6800/1132739058 conn(0x7f4b4c06dfb0 legacy=0x7f4b4c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 10 monclient: shutdown 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 >> v1:172.21.15.110:6789/0 conn(0x7f4b6014be90 legacy=0x7f4b601d7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 shutdown_connections 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.873+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 >> 172.21.15.110:0/4004108207 conn(0x7f4b600617a0 msgr2=0x7f4b60061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:44.874 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.874+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 shutdown_connections 2024-03-25T19:48:44.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.874+0000 7f4b6590e640 1 -- 172.21.15.110:0/4004108207 wait complete. 2024-03-25T19:48:44.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:44.874+0000 7f4b6590e640 1 librados: shutdown 2024-03-25T19:48:44.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:51.691 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:51.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.797+0000 7ff5f1414640 10 monclient: get_monmap_and_config 2024-03-25T19:48:51.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.797+0000 7ff5f1414640 10 monclient: build_initial_monmap 2024-03-25T19:48:51.797 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.797+0000 7ff5f1414640 10 monclient: monmap: 2024-03-25T19:48:51.797 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:48:51.798808+0000 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:48:51.798808+0000 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:48:51.798 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:51.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.798+0000 7ff5f1414640 1 Processor -- start 2024-03-25T19:48:51.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.798+0000 7ff5f1414640 1 -- start start 2024-03-25T19:48:51.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.798+0000 7ff5f1414640 10 monclient: init 2024-03-25T19:48:51.799 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.798+0000 7ff5f1414640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.798+0000 7ff5f1414640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5f1414640 10 monclient(hunting): picked mon.noname-a con 0x7ff5ec0a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5f1414640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ec13f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5f1414640 10 monclient(hunting): picked mon.noname-b con 0x7ff5ec1430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5f1414640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ec142520 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5f1414640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ec0b28a0 con 0x7ff5ec13f710 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5f1414640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ec0b2a30 con 0x7ff5ec1430c0 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5f1414640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5f1414640 10 monclient(hunting): authenticate will time out at 4702.939910s 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5eb7fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7ff5ec1430c0 0x7ff5ec14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:52692/0 (socket says 172.21.15.110:52692) 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5ea7fc640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff5ec13f710 0x7ff5ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:60026/0 (socket says 172.21.15.110:60026) 2024-03-25T19:48:51.800 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.799+0000 7ff5eb7fe640 1 -- 172.21.15.110:0/1277200418 learned_addr learned my addr 172.21.15.110:0/1277200418 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:51.822 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.823+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741729061 0 0) 0x7ff5ec142520 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.823+0000 7ff5e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.823+0000 7ff5e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.823+0000 7ff5e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.823+0000 7ff5e9ffb640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.823+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5d8003520 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1117031829 0 0) 0x7ff5d8003520 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 >> v1:172.21.15.110:6790/0 conn(0x7ff5ec1430c0 legacy=0x7ff5ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 >> v1:172.21.15.178:6789/0 conn(0x7ff5ec13f710 legacy=0x7ff5ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 1 monclient: found mon.noname-a 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ec0b44c0 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff5e4001a10 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5e9ffb640 10 monclient: dump: 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5f1414640 5 monclient: authenticate success, global_id 4599 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.835+0000 7ff5f1414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.839+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5e4001b70 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.839+0000 7ff5e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.839+0000 7ff5e9ffb640 1 -- 172.21.15.110:0/1277200418 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff5e4001f80 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.839+0000 7ff5f1414640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.839+0000 7ff5f1414640 4 set_mon_vals no callback set 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.839+0000 7ff5e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.839+0000 7ff5e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.839+0000 7ff5e9ffb640 10 monclient: dump: 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:51.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.840+0000 7ff5f1414640 10 monclient: shutdown 2024-03-25T19:48:51.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.840+0000 7ff5f1414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:51.842 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.840+0000 7ff5f1414640 1 -- 172.21.15.110:0/1277200418 >> v1:172.21.15.110:6789/0 conn(0x7ff5ec0a98f0 legacy=0x7ff5ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.840+0000 7ff5f1414640 1 -- 172.21.15.110:0/1277200418 shutdown_connections 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.840+0000 7ff5f1414640 1 -- 172.21.15.110:0/1277200418 >> 172.21.15.110:0/1277200418 conn(0x7ff5ec0b7ea0 msgr2=0x7ff5ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.840+0000 7ff5f1414640 1 -- 172.21.15.110:0/1277200418 shutdown_connections 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.840+0000 7ff5f1414640 1 -- 172.21.15.110:0/1277200418 wait complete. 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.841+0000 7ff5f1414640 10 monclient: build_initial_monmap 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.841+0000 7ff5f1414640 10 monclient: monmap: 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:51.843 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.841+0000 7ff5f1414640 1 librados: starting msgr at 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.841+0000 7ff5f1414640 1 librados: starting objecter 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.841+0000 7ff5f1414640 1 Processor -- start 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.841+0000 7ff5f1414640 1 -- start start 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.841+0000 7ff5f1414640 1 librados: setting wanted keys 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 1 librados: calling monclient init 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 10 monclient: init 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 10 monclient(hunting): picked mon.noname-b con 0x7ff5ec0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 10 monclient(hunting): picked mon.noname-a con 0x7ff5ec13f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:51.844 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 10 monclient(hunting): picked mon.noname-c con 0x7ff5ec1430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:51.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ec0b7990 con 0x7ff5ec13f710 2024-03-25T19:48:51.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ec137080 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ec1351c0 con 0x7ff5ec1430c0 2024-03-25T19:48:51.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:51.845 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.842+0000 7ff5f1414640 10 monclient(hunting): authenticate will time out at 4702.983402s 2024-03-25T19:48:51.850 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.851+0000 7ff5ea7fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff5ec13f710 0x7ff5ec1db110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:41780/0 (socket says 172.21.15.110:41780) 2024-03-25T19:48:51.850 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.851+0000 7ff5ea7fc640 1 -- 172.21.15.110:0/1754256174 learned_addr learned my addr 172.21.15.110:0/1754256174 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:51.851 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.852+0000 7ff5e37fe640 10 client.?.objecter ms_handle_connect 0x7ff5ec13f710 2024-03-25T19:48:51.851 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.852+0000 7ff5e37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.862 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.862+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907197276 0 0) 0x7ff5ec0b7990 con 0x7ff5ec13f710 2024-03-25T19:48:51.862 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.862+0000 7ff5e37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.862 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.862+0000 7ff5e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.862 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.862+0000 7ff5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.862 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.862+0000 7ff5e37fe640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:48:51.862 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.862+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5cc004eb0 con 0x7ff5ec13f710 2024-03-25T19:48:51.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.872+0000 7ff5e37fe640 10 client.?.objecter ms_handle_connect 0x7ff5ec0a98f0 2024-03-25T19:48:51.872 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.872+0000 7ff5e37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.879+0000 7ff5e37fe640 10 client.?.objecter ms_handle_connect 0x7ff5ec1430c0 2024-03-25T19:48:51.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.879+0000 7ff5e37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.886+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975214811 0 0) 0x7ff5ec137080 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.886+0000 7ff5e37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.886+0000 7ff5e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.886+0000 7ff5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.886+0000 7ff5e37fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:48:51.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.886+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ec0b7990 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.888+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3310446563 0 0) 0x7ff5cc004eb0 con 0x7ff5ec13f710 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.888+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5ec137080 con 0x7ff5ec13f710 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.888+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff5d4001970 con 0x7ff5ec13f710 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.888+0000 7ff5e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.888+0000 7ff5e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.888+0000 7ff5e37fe640 10 monclient(hunting): dump: 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.888 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:51.889 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:51.889 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:51.889 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:51.889 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:51.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.888+0000 7ff5e37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:51.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.893+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890733008 0 0) 0x7ff5ec1351c0 con 0x7ff5ec1430c0 2024-03-25T19:48:51.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.893+0000 7ff5e37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.893+0000 7ff5e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.893+0000 7ff5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.893+0000 7ff5e37fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:48:51.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.893+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5cc004eb0 con 0x7ff5ec1430c0 2024-03-25T19:48:51.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.896+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3188150724 0 0) 0x7ff5ec0b7990 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.896+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5ec1351c0 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.896+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff5e4001f00 con 0x7ff5ec0a98f0 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.896+0000 7ff5e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.896+0000 7ff5e37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.896+0000 7ff5e37fe640 10 monclient(hunting): dump: 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:51.896 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:51.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 205953744 0 0) 0x7ff5ec137080 con 0x7ff5ec13f710 2024-03-25T19:48:51.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 >> v1:172.21.15.110:6790/0 conn(0x7ff5ec1430c0 legacy=0x7ff5ec1e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 >> v1:172.21.15.178:6789/0 conn(0x7ff5ec0a98f0 legacy=0x7ff5ec1daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 1 monclient: found mon.a 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ec1354e0 con 0x7ff5ec13f710 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 5 monclient: authenticate success, global_id 4602 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 10 monclient: _renew_subs 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ec1357b0 con 0x7ff5ec13f710 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 10 monclient: _renew_subs 2024-03-25T19:48:51.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:51.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ec13b7f0 con 0x7ff5ec13f710 2024-03-25T19:48:51.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.899+0000 7ff5f1414640 1 librados: init done 2024-03-25T19:48:51.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.900+0000 7ff5f1414640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:51.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.900+0000 7ff5f1414640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:51.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.900+0000 7ff5f1414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:51.901 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.900+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ec0b76f0 con 0x7ff5ec13f710 2024-03-25T19:48:51.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.905+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5d40047e0 con 0x7ff5ec13f710 2024-03-25T19:48:51.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.905+0000 7ff5e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:51.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:51.905+0000 7ff5e97fa640 4 set_mon_vals no callback set 2024-03-25T19:48:52.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.208+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7ff5d4004af0 con 0x7ff5ec13f710 2024-03-25T19:48:52.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.208+0000 7ff5e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:52.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.208+0000 7ff5e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:52.208 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.208+0000 7ff5e37fe640 10 monclient: dump: 2024-03-25T19:48:52.208 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:52.209 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:52.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.209+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7ff5d401aac0 con 0x7ff5ec13f710 2024-03-25T19:48:52.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.209+0000 7ff5eaffd640 10 monclient: get_auth_request con 0x7ff5cc06df20 auth_method 0 2024-03-25T19:48:52.210 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.210+0000 7ff5e37fe640 10 client.4602.objecter ms_handle_connect 0x7ff5cc06df20 2024-03-25T19:48:52.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.216+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7ff5d40548a0 con 0x7ff5ec13f710 2024-03-25T19:48:52.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.216+0000 7ff5e37fe640 10 client.4602.objecter ms_dispatch 0x7ff5ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:52.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.216+0000 7ff5e37fe640 3 client.4602.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:52.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.216+0000 7ff5e37fe640 3 client.4602.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:52.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.216+0000 7ff5e37fe640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-25T19:48:52.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.216+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5d40501e0 con 0x7ff5ec13f710 2024-03-25T19:48:52.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.216+0000 7ff5e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:52.216 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.216+0000 7ff5e37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:52.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.466+0000 7ff5f1414640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:52.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.466+0000 7ff5f1414640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:52.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.466+0000 7ff5f1414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:48:52.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.466+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5ec074cc0 con 0x7ff5ec13f710 2024-03-25T19:48:52.493 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.494+0000 7ff5e37fe640 1 -- 172.21.15.110:0/1754256174 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7ff5d40522e0 con 0x7ff5ec13f710 2024-03-25T19:48:52.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.494+0000 7ff5e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:52.494 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.494+0000 7ff5e37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:52.494 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:48:52.494 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 10 librados: watch_flush enter 2024-03-25T19:48:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 10 librados: watch_flush exit 2024-03-25T19:48:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-25T19:48:52.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 10 client.4602.objecter successfully canceled tick 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 >> v1:172.21.15.178:6800/1132739058 conn(0x7ff5cc06df20 legacy=0x7ff5cc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 10 monclient: shutdown 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 >> v1:172.21.15.110:6789/0 conn(0x7ff5ec13f710 legacy=0x7ff5ec1db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 shutdown_connections 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 >> 172.21.15.110:0/1754256174 conn(0x7ff5ec0b7ea0 msgr2=0x7ff5ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 shutdown_connections 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 1 -- 172.21.15.110:0/1754256174 wait complete. 2024-03-25T19:48:52.497 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:52.496+0000 7ff5f1414640 1 librados: shutdown 2024-03-25T19:48:52.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:59.315 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:59.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.432+0000 7faeecb77640 10 monclient: get_monmap_and_config 2024-03-25T19:48:59.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.432+0000 7faeecb77640 10 monclient: build_initial_monmap 2024-03-25T19:48:59.432 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.432+0000 7faeecb77640 10 monclient: monmap: 2024-03-25T19:48:59.432 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:48:59.433767+0000 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:48:59.433767+0000 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:59.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.433+0000 7faeecb77640 1 Processor -- start 2024-03-25T19:48:59.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.433+0000 7faeecb77640 1 -- start start 2024-03-25T19:48:59.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.433+0000 7faeecb77640 10 monclient: init 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 10 monclient(hunting): picked mon.noname-a con 0x7faee80b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 10 monclient(hunting): picked mon.noname-c con 0x7faee80b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 10 monclient(hunting): picked mon.noname-b con 0x7faee814be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee80af940 con 0x7faee80b4d60 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee80b0a70 con 0x7faee80b1730 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee80b0d50 con 0x7faee814be90 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:59.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.434+0000 7faeecb77640 10 monclient(hunting): authenticate will time out at 4710.575070s 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.442+0000 7faee6d9d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7faee814be90 0x7faee814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:57640/0 (socket says 172.21.15.110:57640) 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.442+0000 7faee6d9d640 1 -- 172.21.15.110:0/1418429402 learned_addr learned my addr 172.21.15.110:0/1418429402 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.450+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866750060 0 0) 0x7faee80af940 con 0x7faee80b4d60 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.450+0000 7faee5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.450+0000 7faee5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.450+0000 7faee5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.450+0000 7faee5d9b640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.450+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed4003520 con 0x7faee80b4d60 2024-03-25T19:48:59.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.454+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694057419 0 0) 0x7faee80b0d50 con 0x7faee814be90 2024-03-25T19:48:59.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.454+0000 7faee5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.454+0000 7faee5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.454+0000 7faee5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.454+0000 7faee5d9b640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:48:59.454 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.454+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee80af940 con 0x7faee814be90 2024-03-25T19:48:59.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.458+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893163695 0 0) 0x7faee80b0a70 con 0x7faee80b1730 2024-03-25T19:48:59.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.458+0000 7faee5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.458+0000 7faee5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.458+0000 7faee5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.458+0000 7faee5d9b640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:48:59.458 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.458+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee80b0d50 con 0x7faee80b1730 2024-03-25T19:48:59.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2085717860 0 0) 0x7faed4003520 con 0x7faee80b4d60 2024-03-25T19:48:59.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 >> v1:172.21.15.110:6790/0 conn(0x7faee814be90 legacy=0x7faee814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.459 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 >> v1:172.21.15.178:6789/0 conn(0x7faee80b1730 legacy=0x7faee80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 1 monclient: found mon.noname-a 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee80b1030 con 0x7faee80b4d60 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faed8001690 con 0x7faee80b4d60 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faee5d9b640 10 monclient: dump: 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:59.460 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:59.461 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:59.461 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:59.461 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:59.461 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:59.461 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:59.461 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:59.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faeecb77640 5 monclient: authenticate success, global_id 4605 2024-03-25T19:48:59.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.459+0000 7faeecb77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.464+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faed80017f0 con 0x7faee80b4d60 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.464+0000 7faee5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.464+0000 7faee5d9b640 1 -- 172.21.15.110:0/1418429402 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faed8001c00 con 0x7faee80b4d60 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.464+0000 7faee5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.464+0000 7faee5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.464+0000 7faee5d9b640 10 monclient: dump: 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:59.464 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:59.465 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:59.465 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:59.465 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:59.465 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:59.465 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:59.465 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.464+0000 7faeecb77640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:59.465 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.464+0000 7faeecb77640 4 set_mon_vals no callback set 2024-03-25T19:48:59.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.465+0000 7faeecb77640 10 monclient: shutdown 2024-03-25T19:48:59.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.465+0000 7faeecb77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:59.466 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.465+0000 7faeecb77640 1 -- 172.21.15.110:0/1418429402 >> v1:172.21.15.110:6789/0 conn(0x7faee80b4d60 legacy=0x7faee813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.465+0000 7faeecb77640 1 -- 172.21.15.110:0/1418429402 shutdown_connections 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.465+0000 7faeecb77640 1 -- 172.21.15.110:0/1418429402 >> 172.21.15.110:0/1418429402 conn(0x7faee80617a0 msgr2=0x7faee8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.465+0000 7faeecb77640 1 -- 172.21.15.110:0/1418429402 shutdown_connections 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.465+0000 7faeecb77640 1 -- 172.21.15.110:0/1418429402 wait complete. 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.466+0000 7faeecb77640 10 monclient: build_initial_monmap 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.466+0000 7faeecb77640 10 monclient: monmap: 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:48:59.467 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:48:59.468 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:59.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.466+0000 7faeecb77640 1 librados: starting msgr at 2024-03-25T19:48:59.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.466+0000 7faeecb77640 1 librados: starting objecter 2024-03-25T19:48:59.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 1 Processor -- start 2024-03-25T19:48:59.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 1 -- start start 2024-03-25T19:48:59.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 1 librados: setting wanted keys 2024-03-25T19:48:59.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 1 librados: calling monclient init 2024-03-25T19:48:59.468 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 10 monclient: init 2024-03-25T19:48:59.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:59.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:59.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 10 monclient(hunting): picked mon.noname-c con 0x7faee80b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:48:59.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 10 monclient(hunting): picked mon.noname-b con 0x7faee80b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:48:59.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 10 monclient(hunting): picked mon.noname-a con 0x7faee814be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:48:59.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee81d77c0 con 0x7faee814be90 2024-03-25T19:48:59.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee8135b90 con 0x7faee80b4d60 2024-03-25T19:48:59.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee81306c0 con 0x7faee80b1730 2024-03-25T19:48:59.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:59.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.467+0000 7faeecb77640 10 monclient(hunting): authenticate will time out at 4710.608718s 2024-03-25T19:48:59.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.472+0000 7faee659c640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7faee80b1730 0x7faee812ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:57688/0 (socket says 172.21.15.110:57688) 2024-03-25T19:48:59.472 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.472+0000 7faee659c640 1 -- 172.21.15.110:0/90843591 learned_addr learned my addr 172.21.15.110:0/90843591 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:48:59.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.473+0000 7faedf7fe640 10 client.?.objecter ms_handle_connect 0x7faee80b1730 2024-03-25T19:48:59.473 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.473+0000 7faedf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:59.477 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.477+0000 7faedf7fe640 10 client.?.objecter ms_handle_connect 0x7faee80b4d60 2024-03-25T19:48:59.477 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.477+0000 7faedf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:59.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.480+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729806354 0 0) 0x7faee81306c0 con 0x7faee80b1730 2024-03-25T19:48:59.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.480+0000 7faedf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.480+0000 7faedf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.480+0000 7faedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.480+0000 7faedf7fe640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:48:59.480 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.480+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec8004eb0 con 0x7faee80b1730 2024-03-25T19:48:59.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.486+0000 7faedf7fe640 10 client.?.objecter ms_handle_connect 0x7faee814be90 2024-03-25T19:48:59.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.486+0000 7faedf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.495+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334855513 0 0) 0x7faec8004eb0 con 0x7faee80b1730 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.495+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faee81306c0 con 0x7faee80b1730 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.495+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faed8001bc0 con 0x7faee80b1730 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.495+0000 7faedf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.495+0000 7faedf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.495+0000 7faedf7fe640 10 monclient(hunting): dump: 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:59.495 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:59.496 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:59.496 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:59.496 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:59.496 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:59.496 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:59.496 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.495+0000 7faedf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:59.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.503+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209822422 0 0) 0x7faee8135b90 con 0x7faee80b4d60 2024-03-25T19:48:59.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.503+0000 7faedf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.503+0000 7faedf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.503+0000 7faedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.503 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.503+0000 7faedf7fe640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:48:59.504 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.503+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec8004eb0 con 0x7faee80b4d60 2024-03-25T19:48:59.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.506+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483110568 0 0) 0x7faee81d77c0 con 0x7faee814be90 2024-03-25T19:48:59.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.506+0000 7faedf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.506+0000 7faedf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.506+0000 7faedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.506+0000 7faedf7fe640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:48:59.506 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.506+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee8135b90 con 0x7faee814be90 2024-03-25T19:48:59.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4291794068 0 0) 0x7faee81306c0 con 0x7faee80b1730 2024-03-25T19:48:59.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 >> v1:172.21.15.178:6789/0 conn(0x7faee80b4d60 legacy=0x7faee81d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 >> v1:172.21.15.110:6789/0 conn(0x7faee814be90 legacy=0x7faee8136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 1 monclient: found mon.c 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8143900 con 0x7faee80b1730 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faedf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 5 monclient: authenticate success, global_id 4619 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 10 monclient: _renew_subs 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee8135e20 con 0x7faee80b1730 2024-03-25T19:48:59.510 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:59.511 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 10 monclient: _renew_subs 2024-03-25T19:48:59.511 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:59.511 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faee81d8540 con 0x7faee80b1730 2024-03-25T19:48:59.511 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.509+0000 7faeecb77640 1 librados: init done 2024-03-25T19:48:59.511 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.510+0000 7faeecb77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:59.511 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.510+0000 7faeecb77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:59.511 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.510+0000 7faeecb77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:48:59.511 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.510+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee8143740 con 0x7faee80b1730 2024-03-25T19:48:59.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.808+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faed80011f0 con 0x7faee80b1730 2024-03-25T19:48:59.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.808+0000 7faedf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:59.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.808+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faed8003b00 con 0x7faee80b1730 2024-03-25T19:48:59.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.808+0000 7faedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:59.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.808+0000 7faedf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:59.808 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.808+0000 7faedf7fe640 10 monclient: dump: 2024-03-25T19:48:59.808 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:48:59.809 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.808+0000 7faee559a640 4 set_mon_vals no callback set 2024-03-25T19:48:59.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.809+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7faed8003d20 con 0x7faee80b1730 2024-03-25T19:48:59.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.810+0000 7faeddd9b640 10 monclient: get_auth_request con 0x7faec806de10 auth_method 0 2024-03-25T19:48:59.810 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.810+0000 7faedf7fe640 10 client.4619.objecter ms_handle_connect 0x7faec806de10 2024-03-25T19:48:59.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.817+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7faed8054980 con 0x7faee80b1730 2024-03-25T19:48:59.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.817+0000 7faedf7fe640 10 client.4619.objecter ms_dispatch 0x7faee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:59.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.817+0000 7faedf7fe640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:59.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.817+0000 7faedf7fe640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:59.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.817+0000 7faedf7fe640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T19:48:59.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.817+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faed80502c0 con 0x7faee80b1730 2024-03-25T19:48:59.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.817+0000 7faedf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:59.817 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:48:59.817+0000 7faedf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:00.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.044+0000 7faeecb77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:00.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.044+0000 7faeecb77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:00.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.044+0000 7faeecb77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:00.045 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.044+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faee80627b0 con 0x7faee80b1730 2024-03-25T19:49:00.075 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.076+0000 7faedf7fe640 1 -- 172.21.15.110:0/90843591 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7faed80523c0 con 0x7faee80b1730 2024-03-25T19:49:00.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.076+0000 7faedf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:00.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.076+0000 7faedf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:00.076 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:49:00.076 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:00.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 10 librados: watch_flush enter 2024-03-25T19:49:00.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 10 librados: watch_flush exit 2024-03-25T19:49:00.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T19:49:00.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 10 client.4619.objecter successfully canceled tick 2024-03-25T19:49:00.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 >> v1:172.21.15.178:6800/1132739058 conn(0x7faec806de10 legacy=0x7faec80702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 10 monclient: shutdown 2024-03-25T19:49:00.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:00.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 >> v1:172.21.15.110:6790/0 conn(0x7faee80b1730 legacy=0x7faee812ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 shutdown_connections 2024-03-25T19:49:00.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 >> 172.21.15.110:0/90843591 conn(0x7faee80617a0 msgr2=0x7faee80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:00.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 shutdown_connections 2024-03-25T19:49:00.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 1 -- 172.21.15.110:0/90843591 wait complete. 2024-03-25T19:49:00.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:00.078+0000 7faeecb77640 1 librados: shutdown 2024-03-25T19:49:00.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:03.589 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:10.873 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.639 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:10.923 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.670 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:10.954 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.704 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:10.959 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.708 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:10.963 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.721 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:11.005 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.727 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:11.011 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.738 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:10.993 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.766 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:11.050 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:06.898 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.011+0000 7f97b6a0e640 10 monclient: get_monmap_and_config 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.011+0000 7f97b6a0e640 10 monclient: build_initial_monmap 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.011+0000 7f97b6a0e640 10 monclient: monmap: 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:49:07.013039+0000 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:49:07.013039+0000 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.011+0000 7f97b6a0e640 1 Processor -- start 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.011+0000 7f97b6a0e640 1 -- start start 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.011+0000 7f97b6a0e640 10 monclient: init 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f97b00b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f97b00b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f97b014be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b00af940 con 0x7f97b00b4d60 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b00b0a70 con 0x7f97b00b1730 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b00b0d50 con 0x7f97b014be90 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:07.014 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.012+0000 7f97b6a0e640 10 monclient(hunting): authenticate will time out at 4718.154447s 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.025+0000 7f97affff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f97b00b4d60 0x7f97b013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33882/0 (socket says 172.21.15.110:33882) 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.025+0000 7f97affff640 1 -- 172.21.15.110:0/311105469 learned_addr learned my addr 172.21.15.110:0/311105469 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:07.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.037+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792116306 0 0) 0x7f97b00b0d50 con 0x7f97b014be90 2024-03-25T19:49:07.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.037+0000 7f97aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.037+0000 7f97aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.037+0000 7f97aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.037+0000 7f97aeffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:49:07.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.037+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9798003520 con 0x7f97b014be90 2024-03-25T19:49:07.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.038+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221312388 0 0) 0x7f97b00af940 con 0x7f97b00b4d60 2024-03-25T19:49:07.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.038+0000 7f97aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.038+0000 7f97aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.038+0000 7f97aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.038+0000 7f97aeffd640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:49:07.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.038+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b00b0d50 con 0x7f97b00b4d60 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642307192 0 0) 0x7f97b00b0a70 con 0x7f97b00b1730 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b00af940 con 0x7f97b00b1730 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1673741411 0 0) 0x7f9798003520 con 0x7f97b014be90 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 >> v1:172.21.15.178:6789/0 conn(0x7f97b00b1730 legacy=0x7f97b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 >> v1:172.21.15.110:6789/0 conn(0x7f97b00b4d60 legacy=0x7f97b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.046 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 1 monclient: found mon.noname-b 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.044+0000 7f97aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b00b1030 con 0x7f97b014be90 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f97a8001a10 con 0x7f97b014be90 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97b6a0e640 5 monclient: authenticate success, global_id 4622 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97b6a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97aeffd640 10 monclient: dump: 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:07.047 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:07.048 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.045+0000 7f97b6a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97a8001b70 con 0x7f97b014be90 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97aeffd640 1 -- 172.21.15.110:0/311105469 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f97a8001f80 con 0x7f97b014be90 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97aeffd640 10 monclient: dump: 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.053 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:07.054 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:07.054 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:07.054 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:07.054 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:07.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97b6a0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:07.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97b6a0e640 4 set_mon_vals no callback set 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97b6a0e640 10 monclient: shutdown 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97b6a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.052+0000 7f97b6a0e640 1 -- 172.21.15.110:0/311105469 >> v1:172.21.15.110:6790/0 conn(0x7f97b014be90 legacy=0x7f97b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.053+0000 7f97b6a0e640 1 -- 172.21.15.110:0/311105469 shutdown_connections 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.053+0000 7f97b6a0e640 1 -- 172.21.15.110:0/311105469 >> 172.21.15.110:0/311105469 conn(0x7f97b00617a0 msgr2=0x7f97b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.053+0000 7f97b6a0e640 1 -- 172.21.15.110:0/311105469 shutdown_connections 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.053+0000 7f97b6a0e640 1 -- 172.21.15.110:0/311105469 wait complete. 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.053+0000 7f97b6a0e640 10 monclient: build_initial_monmap 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.053+0000 7f97b6a0e640 10 monclient: monmap: 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:07.057 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 1 librados: starting msgr at 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 1 librados: starting objecter 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 1 Processor -- start 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 1 -- start start 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 1 librados: setting wanted keys 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 1 librados: calling monclient init 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 10 monclient: init 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f97b00b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.054+0000 7f97b6a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f97b00b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:07.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.055+0000 7f97b6a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f97b014be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:07.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.055+0000 7f97b6a0e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b01364f0 con 0x7f97b014be90 2024-03-25T19:49:07.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.055+0000 7f97b6a0e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b0137040 con 0x7f97b00b1730 2024-03-25T19:49:07.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.055+0000 7f97b6a0e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b0137320 con 0x7f97b00b4d60 2024-03-25T19:49:07.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.055+0000 7f97b6a0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:07.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.055+0000 7f97b6a0e640 10 monclient(hunting): authenticate will time out at 4718.196976s 2024-03-25T19:49:07.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.068+0000 7f97affff640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f97b00b1730 0x7f97b0135de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:45312/0 (socket says 172.21.15.110:45312) 2024-03-25T19:49:07.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.068+0000 7f97affff640 1 -- 172.21.15.110:0/3498618684 learned_addr learned my addr 172.21.15.110:0/3498618684 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:07.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.068+0000 7f97acff9640 10 client.?.objecter ms_handle_connect 0x7f97b00b1730 2024-03-25T19:49:07.069 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.068+0000 7f97acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:07.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.075+0000 7f97acff9640 10 client.?.objecter ms_handle_connect 0x7f97b014be90 2024-03-25T19:49:07.076 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.075+0000 7f97acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.076+0000 7f97acff9640 10 client.?.objecter ms_handle_connect 0x7f97b00b4d60 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.076+0000 7f97acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:07.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.079+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254427443 0 0) 0x7f97b0137040 con 0x7f97b00b1730 2024-03-25T19:49:07.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.080+0000 7f97acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.080 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.080+0000 7f97acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.080+0000 7f97acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.080+0000 7f97acff9640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:49:07.081 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.080+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c004eb0 con 0x7f97b00b1730 2024-03-25T19:49:07.089 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.088+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107127203 0 0) 0x7f97b01364f0 con 0x7f97b014be90 2024-03-25T19:49:07.089 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.088+0000 7f97acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.089 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.088+0000 7f97acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.089 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.088+0000 7f97acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.089 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.088+0000 7f97acff9640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:49:07.089 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.088+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b0137040 con 0x7f97b014be90 2024-03-25T19:49:07.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.097+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1322679973 0 0) 0x7f979c004eb0 con 0x7f97b00b1730 2024-03-25T19:49:07.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.097+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97b01364f0 con 0x7f97b00b1730 2024-03-25T19:49:07.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.097+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f97a00019f0 con 0x7f97b00b1730 2024-03-25T19:49:07.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.097+0000 7f97acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.097+0000 7f97acff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:07.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.097+0000 7f97acff9640 10 monclient(hunting): dump: 2024-03-25T19:49:07.098 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:07.098 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:07.099 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.097+0000 7f97acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:07.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.102+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363010513 0 0) 0x7f97b0137320 con 0x7f97b00b4d60 2024-03-25T19:49:07.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.102+0000 7f97acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.102+0000 7f97acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.102+0000 7f97acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.102+0000 7f97acff9640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:49:07.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.102+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c004eb0 con 0x7f97b00b4d60 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.108+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 575545577 0 0) 0x7f97b0137040 con 0x7f97b014be90 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.108+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97b0137320 con 0x7f97b014be90 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.108+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f97a8001e80 con 0x7f97b014be90 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.108+0000 7f97acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.108+0000 7f97acff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.108+0000 7f97acff9640 10 monclient(hunting): dump: 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:07.109 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:07.110 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:07.110 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:07.110 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:07.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1626297501 0 0) 0x7f97b01364f0 con 0x7f97b00b1730 2024-03-25T19:49:07.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 >> v1:172.21.15.110:6790/0 conn(0x7f97b00b4d60 legacy=0x7f97b012ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 >> v1:172.21.15.110:6789/0 conn(0x7f97b014be90 legacy=0x7f97b01e3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.111 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 1 monclient: found mon.b 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b013e1f0 con 0x7f97b00b1730 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 10 monclient: _renew_subs 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b013e3e0 con 0x7f97b00b1730 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 10 monclient: _renew_subs 2024-03-25T19:49:07.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:07.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b012f5d0 con 0x7f97b00b1730 2024-03-25T19:49:07.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.109+0000 7f97b6a0e640 1 librados: init done 2024-03-25T19:49:07.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.111+0000 7f97b6a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:07.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.111+0000 7f97b6a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:07.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.111+0000 7f97b6a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:07.113 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.111+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b00b2270 con 0x7f97b00b1730 2024-03-25T19:49:07.326 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.325+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97a0001c50 con 0x7f97b00b1730 2024-03-25T19:49:07.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.325+0000 7f97acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:07.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.325+0000 7f97ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:49:07.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.367+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f97a0004c30 con 0x7f97b00b1730 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.367+0000 7f97acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.367+0000 7f97acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.367+0000 7f97acff9640 10 monclient: dump: 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:07.368 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:07.375 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.375+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f97a001ac00 con 0x7f97b00b1730 2024-03-25T19:49:07.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.375+0000 7f97af7fe640 10 monclient: get_auth_request con 0x7f979c06dfb0 auth_method 0 2024-03-25T19:49:07.376 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.376+0000 7f97acff9640 10 client.4621.objecter ms_handle_connect 0x7f979c06dfb0 2024-03-25T19:49:07.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.378+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f97a00548a0 con 0x7f97b00b1730 2024-03-25T19:49:07.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.378+0000 7f97acff9640 10 client.4621.objecter ms_dispatch 0x7f97b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:07.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.378+0000 7f97acff9640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:07.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.378+0000 7f97acff9640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:07.379 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.378+0000 7f97acff9640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T19:49:07.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.380+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97a00501e0 con 0x7f97b00b1730 2024-03-25T19:49:07.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.380+0000 7f97acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:07.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.380+0000 7f97acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:07.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.625+0000 7f97b6a0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:07.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.625+0000 7f97b6a0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:07.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.625+0000 7f97b6a0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:07.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.625+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97b00627b0 con 0x7f97b00b1730 2024-03-25T19:49:07.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.650+0000 7f97acff9640 1 -- 172.21.15.110:0/3498618684 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f97a00522e0 con 0x7f97b00b1730 2024-03-25T19:49:07.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.650+0000 7f97acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:07.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.650+0000 7f97acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:07.651 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:49:07.651 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:07.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 10 librados: watch_flush enter 2024-03-25T19:49:07.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 10 librados: watch_flush exit 2024-03-25T19:49:07.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 10 client.4621.objecter successfully canceled tick 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 >> v1:172.21.15.178:6800/1132739058 conn(0x7f979c06dfb0 legacy=0x7f979c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 10 monclient: shutdown 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 >> v1:172.21.15.178:6789/0 conn(0x7f97b00b1730 legacy=0x7f97b0135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 shutdown_connections 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 >> 172.21.15.110:0/3498618684 conn(0x7f97b00617a0 msgr2=0x7f97b013c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 shutdown_connections 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 1 -- 172.21.15.110:0/3498618684 wait complete. 2024-03-25T19:49:07.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:07.653+0000 7f97b6a0e640 1 librados: shutdown 2024-03-25T19:49:07.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:14.473 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:14.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.586+0000 7f81af53d640 10 monclient: get_monmap_and_config 2024-03-25T19:49:14.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.586+0000 7f81af53d640 10 monclient: build_initial_monmap 2024-03-25T19:49:14.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.586+0000 7f81af53d640 10 monclient: monmap: 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:49:14.587784+0000 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:49:14.587784+0000 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:49:14.587 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:14.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.587+0000 7f81af53d640 1 Processor -- start 2024-03-25T19:49:14.588 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.587+0000 7f81af53d640 1 -- start start 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.587+0000 7f81af53d640 10 monclient: init 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 10 monclient(hunting): picked mon.noname-c con 0x7f81a80ae6e0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 10 monclient(hunting): picked mon.noname-b con 0x7f81a806f080 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 10 monclient(hunting): picked mon.noname-a con 0x7f81a80704e0 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a80718d0 con 0x7f81a80704e0 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a806fa30 con 0x7f81a80ae6e0 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a8070090 con 0x7f81a806f080 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:14.589 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.588+0000 7f81af53d640 10 monclient(hunting): authenticate will time out at 4725.729396s 2024-03-25T19:49:14.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.604+0000 7f81ad2d9640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f81a80ae6e0 0x7f81a813c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:41186/0 (socket says 172.21.15.110:41186) 2024-03-25T19:49:14.604 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.604+0000 7f81ad2d9640 1 -- 172.21.15.110:0/3563210351 learned_addr learned my addr 172.21.15.110:0/3563210351 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:14.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.617+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124815744 0 0) 0x7f81a806fa30 con 0x7f81a80ae6e0 2024-03-25T19:49:14.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.617+0000 7f819ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.617+0000 7f819ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.617+0000 7f819ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.617+0000 7f819ffff640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:49:14.617 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.617+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8194003520 con 0x7f81a80ae6e0 2024-03-25T19:49:14.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.620+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920191275 0 0) 0x7f81a80718d0 con 0x7f81a80704e0 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.620+0000 7f819ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.620+0000 7f819ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.620+0000 7f819ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.620+0000 7f819ffff640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.620+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a806fa30 con 0x7f81a80704e0 2024-03-25T19:49:14.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.622+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564884943 0 0) 0x7f81a8070090 con 0x7f81a806f080 2024-03-25T19:49:14.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.622+0000 7f819ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.622+0000 7f819ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.622+0000 7f819ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.622+0000 7f819ffff640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:49:14.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.622+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a80718d0 con 0x7f81a806f080 2024-03-25T19:49:14.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289699101 0 0) 0x7f81a806fa30 con 0x7f81a80704e0 2024-03-25T19:49:14.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 >> v1:172.21.15.110:6790/0 conn(0x7f81a806f080 legacy=0x7f81a8071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 >> v1:172.21.15.178:6789/0 conn(0x7f81a80ae6e0 legacy=0x7f81a813c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 1 monclient: found mon.noname-a 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a806f510 con 0x7f81a80704e0 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 10 monclient: _finish_auth 0 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81a00019f0 con 0x7f81a80704e0 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f819ffff640 10 monclient: dump: 2024-03-25T19:49:14.634 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f81af53d640 5 monclient: authenticate success, global_id 4617 2024-03-25T19:49:14.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.633+0000 7f81af53d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.638+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a0001b50 con 0x7f81a80704e0 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.638+0000 7f819ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.638+0000 7f819ffff640 1 -- 172.21.15.110:0/3563210351 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81a0001f60 con 0x7f81a80704e0 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.638+0000 7f81af53d640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.638+0000 7f81af53d640 4 set_mon_vals no callback set 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.638+0000 7f819ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.638+0000 7f819ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.638+0000 7f819ffff640 10 monclient: dump: 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:14.638 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:14.639 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:14.639 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:14.639 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:14.639 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:14.639 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.639+0000 7f81af53d640 10 monclient: shutdown 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.639+0000 7f81af53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.639+0000 7f81af53d640 1 -- 172.21.15.110:0/3563210351 >> v1:172.21.15.110:6789/0 conn(0x7f81a80704e0 legacy=0x7f81a814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.639+0000 7f81af53d640 1 -- 172.21.15.110:0/3563210351 shutdown_connections 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.639+0000 7f81af53d640 1 -- 172.21.15.110:0/3563210351 >> 172.21.15.110:0/3563210351 conn(0x7f81a806b630 msgr2=0x7f81a80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.639+0000 7f81af53d640 1 -- 172.21.15.110:0/3563210351 shutdown_connections 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.639+0000 7f81af53d640 1 -- 172.21.15.110:0/3563210351 wait complete. 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 10 monclient: build_initial_monmap 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 10 monclient: monmap: 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 1 librados: starting msgr at 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 1 librados: starting objecter 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 1 Processor -- start 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 1 -- start start 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 1 librados: setting wanted keys 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 1 librados: calling monclient init 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.640+0000 7f81af53d640 10 monclient: init 2024-03-25T19:49:14.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 10 monclient(hunting): picked mon.noname-a con 0x7f81a80704e0 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 10 monclient(hunting): picked mon.noname-c con 0x7f81a80ae6e0 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 10 monclient(hunting): picked mon.noname-b con 0x7f81a813ebc0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a8131b90 con 0x7f81a80704e0 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a813f940 con 0x7f81a813ebc0 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a813fc20 con 0x7f81a80ae6e0 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:14.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.641+0000 7f81af53d640 10 monclient(hunting): authenticate will time out at 4725.782343s 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.650+0000 7f81ad2d9640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f81a80704e0 0x7f81a813e4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51958/0 (socket says 172.21.15.110:51958) 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.650+0000 7f81ad2d9640 1 -- 172.21.15.110:0/57542213 learned_addr learned my addr 172.21.15.110:0/57542213 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:14.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.651+0000 7f819dffb640 10 client.?.objecter ms_handle_connect 0x7f81a80704e0 2024-03-25T19:49:14.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.651+0000 7f819dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:14.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.669+0000 7f819dffb640 10 client.?.objecter ms_handle_connect 0x7f81a813ebc0 2024-03-25T19:49:14.668 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.669+0000 7f819dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:14.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.670+0000 7f819dffb640 10 client.?.objecter ms_handle_connect 0x7f81a80ae6e0 2024-03-25T19:49:14.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.670+0000 7f819dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:14.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.672+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689420005 0 0) 0x7f81a8131b90 con 0x7f81a80704e0 2024-03-25T19:49:14.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.672+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.672+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.672+0000 7f819dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.672+0000 7f819dffb640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:49:14.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.672+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c004eb0 con 0x7f81a80704e0 2024-03-25T19:49:14.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.682+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1080954956 0 0) 0x7f818c004eb0 con 0x7f81a80704e0 2024-03-25T19:49:14.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.682+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81a8131b90 con 0x7f81a80704e0 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.682+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81a4001830 con 0x7f81a80704e0 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.682+0000 7f819dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.682+0000 7f819dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.682+0000 7f819dffb640 10 monclient(hunting): dump: 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:14.682 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.682+0000 7f819dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.683+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812382852 0 0) 0x7f81a813f940 con 0x7f81a813ebc0 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.683+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.683+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.683+0000 7f819dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.683+0000 7f819dffb640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.683+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c004eb0 con 0x7f81a813ebc0 2024-03-25T19:49:14.687 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.688+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131370546 0 0) 0x7f81a813fc20 con 0x7f81a80ae6e0 2024-03-25T19:49:14.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.688+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.688+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.688+0000 7f819dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.688+0000 7f819dffb640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:49:14.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.688+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a813f940 con 0x7f81a80ae6e0 2024-03-25T19:49:14.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3850544901 0 0) 0x7f81a8131b90 con 0x7f81a80704e0 2024-03-25T19:49:14.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.110:6790/0 conn(0x7f81a80ae6e0 legacy=0x7f81a8130d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.178:6789/0 conn(0x7f81a813ebc0 legacy=0x7f81a8131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 1 monclient: found mon.a 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a813df70 con 0x7f81a80704e0 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f819dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 5 monclient: authenticate success, global_id 4620 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 10 monclient: _renew_subs 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a81da8a0 con 0x7f81a80704e0 2024-03-25T19:49:14.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 10 client.4620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:14.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 10 monclient: _renew_subs 2024-03-25T19:49:14.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:14.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a8138100 con 0x7f81a80704e0 2024-03-25T19:49:14.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.700+0000 7f81af53d640 1 librados: init done 2024-03-25T19:49:14.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.701+0000 7f81af53d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:14.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.701+0000 7f81af53d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:14.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.701+0000 7f81af53d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:14.703 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.701+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a813fdb0 con 0x7f81a80704e0 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f81ad2d9640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.110:6789/0 conn(0x7f81a80704e0 legacy=0x7f81a813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f81ad2d9640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.110:6789/0 conn(0x7f81a80704e0 legacy=0x7f81a813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f81ad2d9640 1 --1- 172.21.15.110:0/57542213 >> v1:172.21.15.110:6789/0 conn(0x7f81a80704e0 0x7f81a813e4b0 :-1 s=OPENED pgs=66466 cs=1 l=1).handle_message read tag failed 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f81ad2d9640 1 --1- 172.21.15.110:0/57542213 >> v1:172.21.15.110:6789/0 conn(0x7f81a80704e0 0x7f81a813e4b0 :-1 s=OPENED pgs=66466 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.110:6789/0 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.110:6789/0 conn(0x7f81a80704e0 legacy=0x7f81a813e4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 10 monclient(hunting): picked mon.c con 0x7f81a80ae6e0 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 10 monclient(hunting): picked mon.a con 0x7f818c006c20 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 10 monclient(hunting): picked mon.b con 0x7f81a813ebc0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f81a8131b90 con 0x7f818c006c20 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f81a813fc20 con 0x7f81a813ebc0 2024-03-25T19:49:14.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f818c013a40 con 0x7f81a80ae6e0 2024-03-25T19:49:14.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.705+0000 7f819dffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:14.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.721+0000 7f819dffb640 10 client.4620.objecter ms_handle_connect 0x7f81a813ebc0 2024-03-25T19:49:14.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.721+0000 7f819dffb640 10 client.4620.objecter resend_mon_ops 2024-03-25T19:49:14.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.721+0000 7f819dffb640 10 client.4620.objecter ms_handle_connect 0x7f81a80ae6e0 2024-03-25T19:49:14.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.721+0000 7f819dffb640 10 client.4620.objecter resend_mon_ops 2024-03-25T19:49:14.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.733+0000 7f819dffb640 10 client.4620.objecter ms_handle_connect 0x7f818c006c20 2024-03-25T19:49:14.732 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.733+0000 7f819dffb640 10 client.4620.objecter resend_mon_ops 2024-03-25T19:49:14.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.734+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574208308 0 0) 0x7f818c013a40 con 0x7f81a80ae6e0 2024-03-25T19:49:14.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.734+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.734+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.734+0000 7f819dffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:49:14.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.734+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f818c013bd0 con 0x7f81a80ae6e0 2024-03-25T19:49:14.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.741+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495119194 0 0) 0x7f81a8131b90 con 0x7f818c006c20 2024-03-25T19:49:14.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.741+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.741+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.741+0000 7f819dffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:49:14.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.741+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f818c013a40 con 0x7f818c006c20 2024-03-25T19:49:14.743 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.744+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915162703 0 0) 0x7f81a813fc20 con 0x7f81a813ebc0 2024-03-25T19:49:14.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.744+0000 7f819dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.744+0000 7f819dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.744+0000 7f819dffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:49:14.744 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.744+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.178:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81a8131b90 con 0x7f81a813ebc0 2024-03-25T19:49:14.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2838507996 0 0) 0x7f818c013bd0 con 0x7f81a80ae6e0 2024-03-25T19:49:14.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.178:6789/0 conn(0x7f81a813ebc0 legacy=0x7f818c00fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.110:6789/0 conn(0x7f818c006c20 legacy=0x7f818c00c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:14.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 1 monclient: found mon.c 2024-03-25T19:49:14.754 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f818c014e20 con 0x7f81a80ae6e0 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a813fc20 con 0x7f81a80ae6e0 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f81a4004a40 con 0x7f81a80ae6e0 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:14.754+0000 7f819dffb640 10 monclient: dump: 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:14.755 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:14.756 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:14.756 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:14.756 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:14.756 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:14.756 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:14.756 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:14.756 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:15.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.060+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a4004ba0 con 0x7f81a80ae6e0 2024-03-25T19:49:15.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.060+0000 7f819dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:15.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.060+0000 7f819f7fe640 4 set_mon_vals no callback set 2024-03-25T19:49:15.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.061+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f81a401ab80 con 0x7f81a80ae6e0 2024-03-25T19:49:15.061 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.062+0000 7f81acad8640 10 monclient: get_auth_request con 0x7f81a80704e0 auth_method 0 2024-03-25T19:49:15.062 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.062+0000 7f819dffb640 10 client.4620.objecter ms_handle_connect 0x7f81a80704e0 2024-03-25T19:49:15.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.068+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f81a4054870 con 0x7f81a80ae6e0 2024-03-25T19:49:15.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.068+0000 7f819dffb640 10 client.4620.objecter ms_dispatch 0x7f81a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:15.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.068+0000 7f819dffb640 3 client.4620.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:15.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.068+0000 7f819dffb640 3 client.4620.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:15.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.068+0000 7f819dffb640 20 client.4620.objecter dump_active .. 0 homeless 2024-03-25T19:49:15.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.068+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81a40501b0 con 0x7f81a80ae6e0 2024-03-25T19:49:15.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.068+0000 7f819dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:15.068 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.068+0000 7f819dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:15.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.304+0000 7f81af53d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:15.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.304+0000 7f81af53d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:15.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.304+0000 7f81af53d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:15.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.304+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81a8074d80 con 0x7f81a80ae6e0 2024-03-25T19:49:15.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.339+0000 7f819dffb640 1 -- 172.21.15.110:0/57542213 <== mon.2 v1:172.21.15.110:6790/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f81a8074d80 con 0x7f81a80ae6e0 2024-03-25T19:49:15.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.339+0000 7f819dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:15.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.339+0000 7f819dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:15.339 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:49:15.339 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.341+0000 7f81af53d640 10 librados: watch_flush enter 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.341+0000 7f81af53d640 10 librados: watch_flush exit 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.341+0000 7f81af53d640 20 client.4620.objecter shutdown clearing up homeless session... 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.341+0000 7f81af53d640 10 client.4620.objecter successfully canceled tick 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.178:6800/1132739058 conn(0x7f81a80704e0 legacy=0x7f818c07bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 10 monclient: shutdown 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 >> v1:172.21.15.110:6790/0 conn(0x7f81a80ae6e0 legacy=0x7f818c006440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:15.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 shutdown_connections 2024-03-25T19:49:15.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 >> 172.21.15.110:0/57542213 conn(0x7f81a806b630 msgr2=0x7f81a80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:15.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 shutdown_connections 2024-03-25T19:49:15.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 1 -- 172.21.15.110:0/57542213 wait complete. 2024-03-25T19:49:15.343 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:15.342+0000 7f81af53d640 1 librados: shutdown 2024-03-25T19:49:15.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:22.160 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:22.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.278+0000 7fa170d56640 10 monclient: get_monmap_and_config 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.278+0000 7fa170d56640 10 monclient: build_initial_monmap 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.278+0000 7fa170d56640 10 monclient: monmap: 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:49:22.279879+0000 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:49:22.279879+0000 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:49:22.279 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.279+0000 7fa170d56640 1 Processor -- start 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.279+0000 7fa170d56640 1 -- start start 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.279+0000 7fa170d56640 10 monclient: init 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 10 monclient(hunting): picked mon.noname-c con 0x7fa16c0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 10 monclient(hunting): picked mon.noname-b con 0x7fa16c13f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 10 monclient(hunting): picked mon.noname-a con 0x7fa16c1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c142520 con 0x7fa16c1430c0 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c0b28a0 con 0x7fa16c0a98f0 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c0b2a30 con 0x7fa16c13f710 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.280+0000 7fa170d56640 10 monclient(hunting): authenticate will time out at 4733.421608s 2024-03-25T19:49:22.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.290+0000 7fa16a59c640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fa16c0a98f0 0x7fa16c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:56620/0 (socket says 172.21.15.110:56620) 2024-03-25T19:49:22.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.290+0000 7fa16a59c640 1 -- 172.21.15.110:0/1970621385 learned_addr learned my addr 172.21.15.110:0/1970621385 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:22.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.301+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502695612 0 0) 0x7fa16c0b28a0 con 0x7fa16c0a98f0 2024-03-25T19:49:22.300 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.301+0000 7fa16959a640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.301+0000 7fa16959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.301+0000 7fa16959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.301+0000 7fa16959a640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:49:22.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.301+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa154003520 con 0x7fa16c0a98f0 2024-03-25T19:49:22.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.309+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923184912 0 0) 0x7fa16c0b2a30 con 0x7fa16c13f710 2024-03-25T19:49:22.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.309+0000 7fa16959a640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.309+0000 7fa16959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.309+0000 7fa16959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.309+0000 7fa16959a640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:49:22.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.309+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16c0b28a0 con 0x7fa16c13f710 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.312+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1577357711 0 0) 0x7fa154003520 con 0x7fa16c0a98f0 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 >> v1:172.21.15.110:6790/0 conn(0x7fa16c13f710 legacy=0x7fa16c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 >> v1:172.21.15.110:6789/0 conn(0x7fa16c1430c0 legacy=0x7fa16c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 10 monclient: _finish_hunting 0 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 1 monclient: found mon.noname-c 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c0b44c0 con 0x7fa16c0a98f0 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 10 monclient: _finish_auth 0 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 10 monclient: _check_auth_tickets 2024-03-25T19:49:22.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa15c001690 con 0x7fa16c0a98f0 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa170d56640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa170d56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa16959a640 10 monclient: dump: 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:22.314 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:22.315 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:22.315 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:22.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.313+0000 7fa170d56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.320+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa15c0017f0 con 0x7fa16c0a98f0 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.320+0000 7fa16959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.320+0000 7fa16959a640 1 -- 172.21.15.110:0/1970621385 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa15c001c00 con 0x7fa16c0a98f0 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.320+0000 7fa16959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.320+0000 7fa16959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.320+0000 7fa16959a640 10 monclient: dump: 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.320+0000 7fa170d56640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:22.321 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.320+0000 7fa170d56640 4 set_mon_vals no callback set 2024-03-25T19:49:22.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.321+0000 7fa170d56640 10 monclient: shutdown 2024-03-25T19:49:22.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.321+0000 7fa170d56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:22.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.321+0000 7fa170d56640 1 -- 172.21.15.110:0/1970621385 >> v1:172.21.15.178:6789/0 conn(0x7fa16c0a98f0 legacy=0x7fa16c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.321+0000 7fa170d56640 1 -- 172.21.15.110:0/1970621385 shutdown_connections 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.321+0000 7fa170d56640 1 -- 172.21.15.110:0/1970621385 >> 172.21.15.110:0/1970621385 conn(0x7fa16c0b7ea0 msgr2=0x7fa16c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.321+0000 7fa170d56640 1 -- 172.21.15.110:0/1970621385 shutdown_connections 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.321+0000 7fa170d56640 1 -- 172.21.15.110:0/1970621385 wait complete. 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.322+0000 7fa170d56640 10 monclient: build_initial_monmap 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.322+0000 7fa170d56640 10 monclient: monmap: 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:49:22.324 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.322+0000 7fa170d56640 1 librados: starting msgr at 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.322+0000 7fa170d56640 1 librados: starting objecter 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 1 Processor -- start 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 1 -- start start 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 1 librados: setting wanted keys 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 1 librados: calling monclient init 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 10 monclient: init 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 10 monclient(hunting): picked mon.noname-c con 0x7fa16c0a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 10 monclient(hunting): picked mon.noname-a con 0x7fa16c13f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 10 monclient(hunting): picked mon.noname-b con 0x7fa16c1430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.323+0000 7fa170d56640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c135730 con 0x7fa16c13f710 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.324+0000 7fa170d56640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c1358c0 con 0x7fa16c1430c0 2024-03-25T19:49:22.325 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.324+0000 7fa170d56640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c135a50 con 0x7fa16c0a98f0 2024-03-25T19:49:22.326 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.324+0000 7fa170d56640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:22.326 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.324+0000 7fa170d56640 10 monclient(hunting): authenticate will time out at 4733.464863s 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.330+0000 7fa169d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa16c13f710 0x7fa16c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34180/0 (socket says 172.21.15.110:34180) 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.330+0000 7fa169d9b640 1 -- 172.21.15.110:0/2059164324 learned_addr learned my addr 172.21.15.110:0/2059164324 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:22.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.335+0000 7fa15affd640 10 client.?.objecter ms_handle_connect 0x7fa16c13f710 2024-03-25T19:49:22.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.335+0000 7fa15affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:22.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.343+0000 7fa15affd640 10 client.?.objecter ms_handle_connect 0x7fa16c1430c0 2024-03-25T19:49:22.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.343+0000 7fa15affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:22.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.348+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162660864 0 0) 0x7fa16c135730 con 0x7fa16c13f710 2024-03-25T19:49:22.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.348+0000 7fa15affd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.348+0000 7fa15affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.348+0000 7fa15affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.348+0000 7fa15affd640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:49:22.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.348+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c004eb0 con 0x7fa16c13f710 2024-03-25T19:49:22.354 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.355+0000 7fa15affd640 10 client.?.objecter ms_handle_connect 0x7fa16c0a98f0 2024-03-25T19:49:22.355 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.355+0000 7fa15affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:22.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.358+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065869415 0 0) 0x7fa16c1358c0 con 0x7fa16c1430c0 2024-03-25T19:49:22.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.358+0000 7fa15affd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.358+0000 7fa15affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.358+0000 7fa15affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.358+0000 7fa15affd640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:49:22.358 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.358+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16c135730 con 0x7fa16c1430c0 2024-03-25T19:49:22.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.363+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2832165917 0 0) 0x7fa14c004eb0 con 0x7fa16c13f710 2024-03-25T19:49:22.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.363+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa16c1358c0 con 0x7fa16c13f710 2024-03-25T19:49:22.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.363+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa160001a10 con 0x7fa16c13f710 2024-03-25T19:49:22.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.363+0000 7fa15affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:22.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.363+0000 7fa15affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:22.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.363+0000 7fa15affd640 10 monclient(hunting): dump: 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:22.364 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.363+0000 7fa15affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:22.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1247277150 0 0) 0x7fa16c1358c0 con 0x7fa16c13f710 2024-03-25T19:49:22.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 >> v1:172.21.15.110:6790/0 conn(0x7fa16c0a98f0 legacy=0x7fa16c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 >> v1:172.21.15.178:6789/0 conn(0x7fa16c1430c0 legacy=0x7fa16c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 1 monclient: found mon.a 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c1d5000 con 0x7fa16c13f710 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 10 monclient: _finish_auth 0 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa15affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 5 monclient: authenticate success, global_id 4626 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 10 monclient: _renew_subs 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16c1d52b0 con 0x7fa16c13f710 2024-03-25T19:49:22.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:22.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 10 monclient: _renew_subs 2024-03-25T19:49:22.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:22.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16c13c900 con 0x7fa16c13f710 2024-03-25T19:49:22.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.371+0000 7fa170d56640 1 librados: init done 2024-03-25T19:49:22.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.372+0000 7fa170d56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:22.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.372+0000 7fa170d56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:22.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.372+0000 7fa170d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:22.373 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.372+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa16c1371c0 con 0x7fa16c13f710 2024-03-25T19:49:22.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.397+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa160001c50 con 0x7fa16c13f710 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.397+0000 7fa15affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.397+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa160003a60 con 0x7fa16c13f710 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.397+0000 7fa15affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.398+0000 7fa15affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.398+0000 7fa15affd640 10 monclient: dump: 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:22.398 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:22.399 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:22.399 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:22.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.398+0000 7fa168d99640 4 set_mon_vals no callback set 2024-03-25T19:49:22.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.398+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fa160003c80 con 0x7fa16c13f710 2024-03-25T19:49:22.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.399+0000 7fa16a59c640 10 monclient: get_auth_request con 0x7fa14c06dd10 auth_method 0 2024-03-25T19:49:22.399 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.399+0000 7fa15affd640 10 client.4626.objecter ms_handle_connect 0x7fa14c06dd10 2024-03-25T19:49:22.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.402+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fa160054860 con 0x7fa16c13f710 2024-03-25T19:49:22.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.402+0000 7fa15affd640 10 client.4626.objecter ms_dispatch 0x7fa16c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:22.402 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.402+0000 7fa15affd640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:22.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.402+0000 7fa15affd640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:22.403 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.402+0000 7fa15affd640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T19:49:22.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.684+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa160050370 con 0x7fa16c13f710 2024-03-25T19:49:22.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.684+0000 7fa15affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:22.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.684+0000 7fa15affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:22.917 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.916+0000 7fa170d56640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:22.917 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.917+0000 7fa170d56640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:22.917 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.917+0000 7fa170d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:49:22.917 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.917+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa16c0627b0 con 0x7fa16c13f710 2024-03-25T19:49:22.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.942+0000 7fa15affd640 1 -- 172.21.15.110:0/2059164324 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fa1600522a0 con 0x7fa16c13f710 2024-03-25T19:49:22.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.942+0000 7fa15affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:22.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.942+0000 7fa15affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:22.942 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:49:22.942 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:22.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 10 librados: watch_flush enter 2024-03-25T19:49:22.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 10 librados: watch_flush exit 2024-03-25T19:49:22.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T19:49:22.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 10 client.4626.objecter successfully canceled tick 2024-03-25T19:49:22.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 >> v1:172.21.15.178:6800/1132739058 conn(0x7fa14c06dd10 legacy=0x7fa14c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.944 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 10 monclient: shutdown 2024-03-25T19:49:22.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:22.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 >> v1:172.21.15.110:6789/0 conn(0x7fa16c13f710 legacy=0x7fa16c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 shutdown_connections 2024-03-25T19:49:22.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 >> 172.21.15.110:0/2059164324 conn(0x7fa16c0b7ea0 msgr2=0x7fa16c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:22.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 shutdown_connections 2024-03-25T19:49:22.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 1 -- 172.21.15.110:0/2059164324 wait complete. 2024-03-25T19:49:22.945 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:22.944+0000 7fa170d56640 1 librados: shutdown 2024-03-25T19:49:22.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:23.678 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:30.962 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.686 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:30.971 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.715 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:31.000 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.735 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:31.019 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.752 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:31.036 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.763 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:31.018 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.767 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:31.022 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.790 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:31.074 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.798 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:31.053 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:29.763 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:29.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.875+0000 7fdd81c06640 10 monclient: get_monmap_and_config 2024-03-25T19:49:29.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.875+0000 7fdd81c06640 10 monclient: build_initial_monmap 2024-03-25T19:49:29.875 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.875+0000 7fdd81c06640 10 monclient: monmap: 2024-03-25T19:49:29.875 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:29.875 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:29.875 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:49:29.876467+0000 2024-03-25T19:49:29.875 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:49:29.876467+0000 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.876+0000 7fdd81c06640 1 Processor -- start 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.876+0000 7fdd81c06640 1 -- start start 2024-03-25T19:49:29.876 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.876+0000 7fdd81c06640 10 monclient: init 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 10 monclient(hunting): picked mon.noname-c con 0x7fdd7c0a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 10 monclient(hunting): picked mon.noname-b con 0x7fdd7c13f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 10 monclient(hunting): picked mon.noname-a con 0x7fdd7c1430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c142520 con 0x7fdd7c1430c0 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c0b28a0 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c0b2a30 con 0x7fdd7c13f710 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:29.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.877+0000 7fdd81c06640 10 monclient(hunting): authenticate will time out at 4741.018709s 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.884+0000 7fdd7b7fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fdd7c0a98f0 0x7fdd7c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:33214/0 (socket says 172.21.15.110:33214) 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.884+0000 7fdd7b7fe640 1 -- 172.21.15.110:0/28124636 learned_addr learned my addr 172.21.15.110:0/28124636 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:29.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.914+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352697480 0 0) 0x7fdd7c0b28a0 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.914+0000 7fdd7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.914+0000 7fdd7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.914+0000 7fdd7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.914+0000 7fdd7a7fc640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:49:29.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.914+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd64003520 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.925+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106414175 0 0) 0x7fdd7c0b2a30 con 0x7fdd7c13f710 2024-03-25T19:49:29.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.925+0000 7fdd7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.925+0000 7fdd7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.925+0000 7fdd7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.925+0000 7fdd7a7fc640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:49:29.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.925+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd7c0b28a0 con 0x7fdd7c13f710 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880130210 0 0) 0x7fdd64003520 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 >> v1:172.21.15.110:6790/0 conn(0x7fdd7c13f710 legacy=0x7fdd7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 >> v1:172.21.15.110:6789/0 conn(0x7fdd7c1430c0 legacy=0x7fdd7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c0b44c0 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdd6c001690 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd7a7fc640 10 monclient: dump: 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd81c06640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.929+0000 7fdd81c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:29.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd6c0017f0 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd7a7fc640 1 -- 172.21.15.110:0/28124636 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdd6c001c00 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd7a7fc640 10 monclient: dump: 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:29.956 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:29.957 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:29.957 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:29.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd81c06640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:29.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd81c06640 4 set_mon_vals no callback set 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd81c06640 10 monclient: shutdown 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd81c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd81c06640 1 -- 172.21.15.110:0/28124636 >> v1:172.21.15.178:6789/0 conn(0x7fdd7c0a98f0 legacy=0x7fdd7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd81c06640 1 -- 172.21.15.110:0/28124636 shutdown_connections 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.956+0000 7fdd81c06640 1 -- 172.21.15.110:0/28124636 >> 172.21.15.110:0/28124636 conn(0x7fdd7c0b7ea0 msgr2=0x7fdd7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.957+0000 7fdd81c06640 1 -- 172.21.15.110:0/28124636 shutdown_connections 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.957+0000 7fdd81c06640 1 -- 172.21.15.110:0/28124636 wait complete. 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.957+0000 7fdd81c06640 10 monclient: build_initial_monmap 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.957+0000 7fdd81c06640 10 monclient: monmap: 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:29.959 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.958+0000 7fdd81c06640 1 librados: starting msgr at 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.958+0000 7fdd81c06640 1 librados: starting objecter 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.958+0000 7fdd81c06640 1 Processor -- start 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.958+0000 7fdd81c06640 1 -- start start 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.958+0000 7fdd81c06640 1 librados: setting wanted keys 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.958+0000 7fdd81c06640 1 librados: calling monclient init 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.958+0000 7fdd81c06640 10 monclient: init 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:29.960 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:29.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 10 monclient(hunting): picked mon.noname-a con 0x7fdd7c0a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:29.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 10 monclient(hunting): picked mon.noname-b con 0x7fdd7c13f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:29.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 10 monclient(hunting): picked mon.noname-c con 0x7fdd7c1430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:29.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c135730 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c1358c0 con 0x7fdd7c13f710 2024-03-25T19:49:29.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c135a50 con 0x7fdd7c1430c0 2024-03-25T19:49:29.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:29.961 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.959+0000 7fdd81c06640 10 monclient(hunting): authenticate will time out at 4741.100081s 2024-03-25T19:49:29.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.972+0000 7fdd7affd640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fdd7c13f710 0x7fdd7c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:33256/0 (socket says 172.21.15.110:33256) 2024-03-25T19:49:29.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.972+0000 7fdd7affd640 1 -- 172.21.15.110:0/638383477 learned_addr learned my addr 172.21.15.110:0/638383477 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:29.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.974+0000 7fdd63fff640 10 client.?.objecter ms_handle_connect 0x7fdd7c13f710 2024-03-25T19:49:29.973 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.974+0000 7fdd63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:29.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.975+0000 7fdd63fff640 10 client.?.objecter ms_handle_connect 0x7fdd7c1430c0 2024-03-25T19:49:29.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.975+0000 7fdd63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:29.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.979+0000 7fdd63fff640 10 client.?.objecter ms_handle_connect 0x7fdd7c0a98f0 2024-03-25T19:49:29.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.979+0000 7fdd63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:29.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.980+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265680228 0 0) 0x7fdd7c135a50 con 0x7fdd7c1430c0 2024-03-25T19:49:29.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.980+0000 7fdd63fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.980+0000 7fdd63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.980+0000 7fdd63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.980+0000 7fdd63fff640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:49:29.980 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.980+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd68004eb0 con 0x7fdd7c1430c0 2024-03-25T19:49:29.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.986+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944733026 0 0) 0x7fdd7c1358c0 con 0x7fdd7c13f710 2024-03-25T19:49:29.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.986+0000 7fdd63fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.986+0000 7fdd63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.986+0000 7fdd63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.986+0000 7fdd63fff640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:49:29.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.986+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd7c135a50 con 0x7fdd7c13f710 2024-03-25T19:49:29.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.987+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087073567 0 0) 0x7fdd7c135730 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.987+0000 7fdd63fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.987+0000 7fdd63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.987+0000 7fdd63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.987+0000 7fdd63fff640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:49:29.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.987+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd7c1358c0 con 0x7fdd7c0a98f0 2024-03-25T19:49:29.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.999+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141485864 0 0) 0x7fdd68004eb0 con 0x7fdd7c1430c0 2024-03-25T19:49:29.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.999+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd7c135730 con 0x7fdd7c1430c0 2024-03-25T19:49:29.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.999+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdd74001b90 con 0x7fdd7c1430c0 2024-03-25T19:49:29.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.999+0000 7fdd63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:29.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.999+0000 7fdd63fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:29.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.999+0000 7fdd63fff640 10 monclient(hunting): dump: 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:30.000 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:29.999+0000 7fdd63fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.002+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470052457 0 0) 0x7fdd7c135a50 con 0x7fdd7c13f710 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.002+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd68004eb0 con 0x7fdd7c13f710 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.002+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdd700019f0 con 0x7fdd7c13f710 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.002+0000 7fdd63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.002+0000 7fdd63fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.002+0000 7fdd63fff640 10 monclient(hunting): dump: 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:30.002 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:30.003 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:30.003 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:30.003 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:30.003 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:30.003 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:30.003 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:30.003 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:30.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2136942202 0 0) 0x7fdd7c135730 con 0x7fdd7c1430c0 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 >> v1:172.21.15.178:6789/0 conn(0x7fdd7c13f710 legacy=0x7fdd7c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 >> v1:172.21.15.110:6789/0 conn(0x7fdd7c0a98f0 legacy=0x7fdd7c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 1 monclient: found mon.c 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c1d5000 con 0x7fdd7c1430c0 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 5 monclient: authenticate success, global_id 4643 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 10 monclient: _renew_subs 2024-03-25T19:49:30.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:30.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7c1d52b0 con 0x7fdd7c1430c0 2024-03-25T19:49:30.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:30.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 10 monclient: _renew_subs 2024-03-25T19:49:30.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:30.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c13c900 con 0x7fdd7c1430c0 2024-03-25T19:49:30.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.017+0000 7fdd81c06640 1 librados: init done 2024-03-25T19:49:30.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.018+0000 7fdd81c06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:30.019 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.018+0000 7fdd81c06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:30.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.018+0000 7fdd81c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:30.020 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.018+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7c13cbb0 con 0x7fdd7c1430c0 2024-03-25T19:49:30.328 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.329+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd74002080 con 0x7fdd7c1430c0 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.329+0000 7fdd63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.329+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fdd74003970 con 0x7fdd7c1430c0 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.329+0000 7fdd63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.329+0000 7fdd63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.329+0000 7fdd63fff640 10 monclient: dump: 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:30.329 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:30.330 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:30.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.329+0000 7fdd79ffb640 4 set_mon_vals no callback set 2024-03-25T19:49:30.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.330+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fdd74003b90 con 0x7fdd7c1430c0 2024-03-25T19:49:30.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.330+0000 7fdd7b7fe640 10 monclient: get_auth_request con 0x7fdd6806ded0 auth_method 0 2024-03-25T19:49:30.330 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.331+0000 7fdd63fff640 10 client.4643.objecter ms_handle_connect 0x7fdd6806ded0 2024-03-25T19:49:30.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.335+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fdd74053790 con 0x7fdd7c1430c0 2024-03-25T19:49:30.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.336+0000 7fdd63fff640 10 client.4643.objecter ms_dispatch 0x7fdd7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:30.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.336+0000 7fdd63fff640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:30.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.336+0000 7fdd63fff640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:30.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.336+0000 7fdd63fff640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-25T19:49:30.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.336+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd7404f0d0 con 0x7fdd7c1430c0 2024-03-25T19:49:30.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.336+0000 7fdd63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:30.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.336+0000 7fdd63fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:30.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.582+0000 7fdd81c06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:30.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.582+0000 7fdd81c06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:30.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.582+0000 7fdd81c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:30.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.582+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd7c1371c0 con 0x7fdd7c1430c0 2024-03-25T19:49:30.618 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.618+0000 7fdd63fff640 1 -- 172.21.15.110:0/638383477 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fdd740511d0 con 0x7fdd7c1430c0 2024-03-25T19:49:30.618 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.618+0000 7fdd63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:30.618 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.618+0000 7fdd63fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:30.618 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:49:30.618 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:30.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 10 librados: watch_flush enter 2024-03-25T19:49:30.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 10 librados: watch_flush exit 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 10 client.4643.objecter successfully canceled tick 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 >> v1:172.21.15.178:6800/1132739058 conn(0x7fdd6806ded0 legacy=0x7fdd68070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 10 monclient: shutdown 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 >> v1:172.21.15.110:6790/0 conn(0x7fdd7c1430c0 legacy=0x7fdd7c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 shutdown_connections 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 >> 172.21.15.110:0/638383477 conn(0x7fdd7c0b7ea0 msgr2=0x7fdd7c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 shutdown_connections 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 1 -- 172.21.15.110:0/638383477 wait complete. 2024-03-25T19:49:30.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:30.620+0000 7fdd81c06640 1 librados: shutdown 2024-03-25T19:49:30.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:37.439 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:37.555 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.555+0000 7fefbae93640 10 monclient: get_monmap_and_config 2024-03-25T19:49:37.555 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.555+0000 7fefbae93640 10 monclient: build_initial_monmap 2024-03-25T19:49:37.555 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.555+0000 7fefbae93640 10 monclient: monmap: 2024-03-25T19:49:37.555 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:37.555 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:37.556 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:49:37.556790+0000 2024-03-25T19:49:37.556 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:49:37.556790+0000 2024-03-25T19:49:37.556 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:37.556 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:37.556 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:37.556 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:49:37.556 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:49:37.556 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.556+0000 7fefbae93640 1 Processor -- start 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.556+0000 7fefbae93640 1 -- start start 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.556+0000 7fefbae93640 10 monclient: init 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 10 monclient(hunting): picked mon.noname-c con 0x7fefb40a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 10 monclient(hunting): picked mon.noname-a con 0x7fefb413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 10 monclient(hunting): picked mon.noname-b con 0x7fefb41430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:37.557 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb4142520 con 0x7fefb413f710 2024-03-25T19:49:37.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb40b28a0 con 0x7fefb40a98f0 2024-03-25T19:49:37.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb40b2a30 con 0x7fefb41430c0 2024-03-25T19:49:37.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:37.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.557+0000 7fefbae93640 10 monclient(hunting): authenticate will time out at 4748.698228s 2024-03-25T19:49:37.566 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.567+0000 7fefb8c2f640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fefb40a98f0 0x7fefb40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:59932/0 (socket says 172.21.15.110:59932) 2024-03-25T19:49:37.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.567+0000 7fefb8c2f640 1 -- 172.21.15.110:0/2613659965 learned_addr learned my addr 172.21.15.110:0/2613659965 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:37.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.594+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515150626 0 0) 0x7fefb4142520 con 0x7fefb413f710 2024-03-25T19:49:37.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.594+0000 7fefab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:37.594 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.594+0000 7fefab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:37.594 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.594+0000 7fefab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:37.594 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.594+0000 7fefab7fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:49:37.594 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.594+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c003520 con 0x7fefb413f710 2024-03-25T19:49:37.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.657+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680539913 0 0) 0x7fefb40b28a0 con 0x7fefb40a98f0 2024-03-25T19:49:37.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.657+0000 7fefab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:37.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.657+0000 7fefab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:37.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.657+0000 7fefab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:37.657 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.657+0000 7fefab7fe640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:49:37.658 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.657+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb4142520 con 0x7fefb40a98f0 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.665+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2167554059 0 0) 0x7fefb4142520 con 0x7fefb40a98f0 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.665+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 >> v1:172.21.15.110:6790/0 conn(0x7fefb41430c0 legacy=0x7fefb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 >> v1:172.21.15.110:6789/0 conn(0x7fefb413f710 legacy=0x7fefb41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 1 monclient: found mon.noname-c 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb40b44c0 con 0x7fefb40a98f0 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fefb0001670 con 0x7fefb40a98f0 2024-03-25T19:49:37.666 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefab7fe640 10 monclient: dump: 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefbae93640 5 monclient: authenticate success, global_id 4642 2024-03-25T19:49:37.667 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.666+0000 7fefbae93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.688+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefb00017d0 con 0x7fefb40a98f0 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.688+0000 7fefab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.688+0000 7fefab7fe640 1 -- 172.21.15.110:0/2613659965 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fefb0001ba0 con 0x7fefb40a98f0 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.688+0000 7fefbae93640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.688+0000 7fefbae93640 4 set_mon_vals no callback set 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.688+0000 7fefab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.688+0000 7fefab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.688+0000 7fefab7fe640 10 monclient: dump: 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:37.688 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:37.689 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:37.689 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:37.689 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:37.689 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:37.689 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:37.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.689+0000 7fefbae93640 10 monclient: shutdown 2024-03-25T19:49:37.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.689+0000 7fefbae93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:37.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.689+0000 7fefbae93640 1 -- 172.21.15.110:0/2613659965 >> v1:172.21.15.178:6789/0 conn(0x7fefb40a98f0 legacy=0x7fefb40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.689+0000 7fefbae93640 1 -- 172.21.15.110:0/2613659965 shutdown_connections 2024-03-25T19:49:37.690 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.689+0000 7fefbae93640 1 -- 172.21.15.110:0/2613659965 >> 172.21.15.110:0/2613659965 conn(0x7fefb40b7ea0 msgr2=0x7fefb40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.689+0000 7fefbae93640 1 -- 172.21.15.110:0/2613659965 shutdown_connections 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.689+0000 7fefbae93640 1 -- 172.21.15.110:0/2613659965 wait complete. 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 10 monclient: build_initial_monmap 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 10 monclient: monmap: 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 1 librados: starting msgr at 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 1 librados: starting objecter 2024-03-25T19:49:37.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 1 Processor -- start 2024-03-25T19:49:37.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 1 -- start start 2024-03-25T19:49:37.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 1 librados: setting wanted keys 2024-03-25T19:49:37.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 1 librados: calling monclient init 2024-03-25T19:49:37.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.690+0000 7fefbae93640 10 monclient: init 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 10 monclient(hunting): picked mon.noname-a con 0x7fefb40a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 10 monclient(hunting): picked mon.noname-c con 0x7fefb413f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 10 monclient(hunting): picked mon.noname-b con 0x7fefb41430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb4139190 con 0x7fefb40a98f0 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb40aba50 con 0x7fefb41430c0 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb40abd30 con 0x7fefb413f710 2024-03-25T19:49:37.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:37.694 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.691+0000 7fefbae93640 10 monclient(hunting): authenticate will time out at 4748.832329s 2024-03-25T19:49:37.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.704+0000 7fefb9430640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fefb41430c0 0x7fefb41e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:60002/0 (socket says 172.21.15.110:60002) 2024-03-25T19:49:37.704 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.705+0000 7fefb9430640 1 -- 172.21.15.110:0/1444927958 learned_addr learned my addr 172.21.15.110:0/1444927958 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:37.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.706+0000 7fefa97fa640 10 client.?.objecter ms_handle_connect 0x7fefb41430c0 2024-03-25T19:49:37.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.706+0000 7fefa97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:37.719 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.720+0000 7fefa97fa640 10 client.?.objecter ms_handle_connect 0x7fefb413f710 2024-03-25T19:49:37.719 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.720+0000 7fefa97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:37.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.721+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691978564 0 0) 0x7fefb40aba50 con 0x7fefb41430c0 2024-03-25T19:49:37.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.721+0000 7fefa97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:37.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.721+0000 7fefa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:37.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.721+0000 7fefa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:37.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.721+0000 7fefa97fa640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:49:37.721 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.721+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa0004eb0 con 0x7fefb41430c0 2024-03-25T19:49:37.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.724+0000 7fefa97fa640 10 client.?.objecter ms_handle_connect 0x7fefb40a98f0 2024-03-25T19:49:37.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.724+0000 7fefa97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.727+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2398959238 0 0) 0x7fefa0004eb0 con 0x7fefb41430c0 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.727+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefb40aba50 con 0x7fefb41430c0 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.727+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fefac001c30 con 0x7fefb41430c0 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.727+0000 7fefa97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.727+0000 7fefa97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.727+0000 7fefa97fa640 10 monclient(hunting): dump: 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:37.727 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:37.728 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:37.728 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:37.728 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:37.728 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:37.728 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:37.728 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:37.728 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:37.728 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.727+0000 7fefa97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:37.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.729+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054122441 0 0) 0x7fefb40abd30 con 0x7fefb413f710 2024-03-25T19:49:37.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.729+0000 7fefa97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:37.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.729+0000 7fefa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:37.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.729+0000 7fefa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:37.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.729+0000 7fefa97fa640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:49:37.729 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.729+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa0004eb0 con 0x7fefb413f710 2024-03-25T19:49:37.733 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.734+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885673522 0 0) 0x7fefb4139190 con 0x7fefb40a98f0 2024-03-25T19:49:37.733 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.734+0000 7fefa97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:37.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.734+0000 7fefa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:37.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.734+0000 7fefa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:37.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.734+0000 7fefa97fa640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:49:37.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.734+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb40abd30 con 0x7fefb40a98f0 2024-03-25T19:49:37.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3342271573 0 0) 0x7fefb40aba50 con 0x7fefb41430c0 2024-03-25T19:49:37.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 >> v1:172.21.15.110:6790/0 conn(0x7fefb413f710 legacy=0x7fefb40ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 >> v1:172.21.15.110:6789/0 conn(0x7fefb40a98f0 legacy=0x7fefb41388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:49:37.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 1 monclient: found mon.b 2024-03-25T19:49:37.740 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4137d20 con 0x7fefb41430c0 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 10 monclient: _finish_auth 0 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefa97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 10 monclient: _renew_subs 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb41364f0 con 0x7fefb41430c0 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 10 monclient: _renew_subs 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb41e39b0 con 0x7fefb41430c0 2024-03-25T19:49:37.741 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.740+0000 7fefbae93640 1 librados: init done 2024-03-25T19:49:37.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.741+0000 7fefbae93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:37.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.741+0000 7fefbae93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:37.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.741+0000 7fefbae93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:37.742 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:37.741+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb40b3ef0 con 0x7fefb41430c0 2024-03-25T19:49:38.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.034+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefac003970 con 0x7fefb41430c0 2024-03-25T19:49:38.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.034+0000 7fefa97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:38.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.034+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fefac003dc0 con 0x7fefb41430c0 2024-03-25T19:49:38.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.034+0000 7fefa97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:38.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.034+0000 7fefa97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.034+0000 7fefa97fa640 10 monclient: dump: 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.034+0000 7fefaaffd640 4 set_mon_vals no callback set 2024-03-25T19:49:38.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.035+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fefac003fe0 con 0x7fefb41430c0 2024-03-25T19:49:38.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.036+0000 7fefb8c2f640 10 monclient: get_auth_request con 0x7fefa006de60 auth_method 0 2024-03-25T19:49:38.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.036+0000 7fefa97fa640 10 client.4645.objecter ms_handle_connect 0x7fefa006de60 2024-03-25T19:49:38.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.040+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fefac054a50 con 0x7fefb41430c0 2024-03-25T19:49:38.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.040+0000 7fefa97fa640 10 client.4645.objecter ms_dispatch 0x7fefb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:38.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.040+0000 7fefa97fa640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:38.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.040+0000 7fefa97fa640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:38.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.040+0000 7fefa97fa640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T19:49:38.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.040+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefac050560 con 0x7fefb41430c0 2024-03-25T19:49:38.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.040+0000 7fefa97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:38.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.040+0000 7fefa97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:38.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.280+0000 7fefbae93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:38.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.280+0000 7fefbae93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:38.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.280+0000 7fefbae93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:38.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.280+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefb4137eb0 con 0x7fefb41430c0 2024-03-25T19:49:38.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.315+0000 7fefa97fa640 1 -- 172.21.15.110:0/1444927958 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fefac052490 con 0x7fefb41430c0 2024-03-25T19:49:38.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.315+0000 7fefa97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:38.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.315+0000 7fefa97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:38.315 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:49:38.315 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.317+0000 7fefbae93640 10 librados: watch_flush enter 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.317+0000 7fefbae93640 10 librados: watch_flush exit 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.317+0000 7fefbae93640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.317+0000 7fefbae93640 10 client.4645.objecter successfully canceled tick 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.317+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 >> v1:172.21.15.178:6800/1132739058 conn(0x7fefa006de60 legacy=0x7fefa0070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.317+0000 7fefbae93640 10 monclient: shutdown 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.317+0000 7fefbae93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.317+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 >> v1:172.21.15.178:6789/0 conn(0x7fefb41430c0 legacy=0x7fefb41e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.318+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 shutdown_connections 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.318+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 >> 172.21.15.110:0/1444927958 conn(0x7fefb40b7ea0 msgr2=0x7fefb4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.318+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 shutdown_connections 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.318+0000 7fefbae93640 1 -- 172.21.15.110:0/1444927958 wait complete. 2024-03-25T19:49:38.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:38.318+0000 7fefbae93640 1 librados: shutdown 2024-03-25T19:49:38.330 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:43.744 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:51.028 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.803 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:51.087 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.835 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:51.119 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.846 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:51.101 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.848 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:51.103 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.862 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:51.146 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.866 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:51.150 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.882 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:09:51.136 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.898 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:51.182 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:45.137 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.247+0000 7f13decc1640 10 monclient: get_monmap_and_config 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.247+0000 7f13decc1640 10 monclient: build_initial_monmap 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.247+0000 7f13decc1640 10 monclient: monmap: 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:49:45.248222+0000 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:49:45.248222+0000 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:49:45.247 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:45.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 1 Processor -- start 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 1 -- start start 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 10 monclient: init 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 10 monclient(hunting): picked mon.noname-c con 0x7f13d80aead0 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 10 monclient(hunting): picked mon.noname-b con 0x7f13d8135c10 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 10 monclient(hunting): picked mon.noname-a con 0x7f13d806f3d0 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d8071be0 con 0x7f13d806f3d0 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d806fb30 con 0x7f13d80aead0 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d806f840 con 0x7f13d8135c10 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:45.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.248+0000 7f13decc1640 10 monclient(hunting): authenticate will time out at 4756.389748s 2024-03-25T19:49:45.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.260+0000 7f13d7fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f13d8135c10 0x7f13d8072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:56248/0 (socket says 172.21.15.110:56248) 2024-03-25T19:49:45.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.260+0000 7f13d7fff640 1 -- 172.21.15.110:0/423563211 learned_addr learned my addr 172.21.15.110:0/423563211 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:45.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575236515 0 0) 0x7f13d806fb30 con 0x7f13d80aead0 2024-03-25T19:49:45.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:45.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:45.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:45.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:49:45.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c8003520 con 0x7f13d80aead0 2024-03-25T19:49:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001584244 0 0) 0x7f13d8071be0 con 0x7f13d806f3d0 2024-03-25T19:49:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:49:45.271 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.270+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d806fb30 con 0x7f13d806f3d0 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1901785688 0 0) 0x7f13c8003520 con 0x7f13d80aead0 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 >> v1:172.21.15.110:6790/0 conn(0x7f13d8135c10 legacy=0x7f13d8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 >> v1:172.21.15.110:6789/0 conn(0x7f13d806f3d0 legacy=0x7f13d8072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 1 monclient: found mon.noname-c 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80b0200 con 0x7f13d80aead0 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:45.284 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13decc1640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13decc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f13cc001690 con 0x7f13d80aead0 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13d77fe640 10 monclient: dump: 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:45.285 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:45.286 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:45.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.284+0000 7f13decc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:45.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.309+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13cc0017f0 con 0x7f13d80aead0 2024-03-25T19:49:45.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.309+0000 7f13d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:45.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.309+0000 7f13d77fe640 1 -- 172.21.15.110:0/423563211 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f13cc001c00 con 0x7f13d80aead0 2024-03-25T19:49:45.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.309+0000 7f13decc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:45.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.309+0000 7f13decc1640 4 set_mon_vals no callback set 2024-03-25T19:49:45.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.309+0000 7f13d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.309+0000 7f13d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.309+0000 7f13d77fe640 10 monclient: dump: 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:45.310 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:45.311 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.310+0000 7f13decc1640 10 monclient: shutdown 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.310+0000 7f13decc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.310+0000 7f13decc1640 1 -- 172.21.15.110:0/423563211 >> v1:172.21.15.178:6789/0 conn(0x7f13d80aead0 legacy=0x7f13d8135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.310+0000 7f13decc1640 1 -- 172.21.15.110:0/423563211 shutdown_connections 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.310+0000 7f13decc1640 1 -- 172.21.15.110:0/423563211 >> 172.21.15.110:0/423563211 conn(0x7f13d80b3e60 msgr2=0x7f13d80b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.310+0000 7f13decc1640 1 -- 172.21.15.110:0/423563211 shutdown_connections 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.310+0000 7f13decc1640 1 -- 172.21.15.110:0/423563211 wait complete. 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.311+0000 7f13decc1640 10 monclient: build_initial_monmap 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.311+0000 7f13decc1640 10 monclient: monmap: 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:49:45.312 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.311+0000 7f13decc1640 1 librados: starting msgr at 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.311+0000 7f13decc1640 1 librados: starting objecter 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.311+0000 7f13decc1640 1 Processor -- start 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 1 -- start start 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 1 librados: setting wanted keys 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 1 librados: calling monclient init 2024-03-25T19:49:45.313 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 10 monclient: init 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 10 monclient(hunting): picked mon.noname-c con 0x7f13d806f3d0 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 10 monclient(hunting): picked mon.noname-a con 0x7f13d80aead0 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 10 monclient(hunting): picked mon.noname-b con 0x7f13d8135c10 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d813cfe0 con 0x7f13d80aead0 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d8136410 con 0x7f13d8135c10 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d813eef0 con 0x7f13d806f3d0 2024-03-25T19:49:45.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:45.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.312+0000 7f13decc1640 10 monclient(hunting): authenticate will time out at 4756.453740s 2024-03-25T19:49:45.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.320+0000 7f13d7fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f13d80aead0 0x7f13d81da0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51064/0 (socket says 172.21.15.110:51064) 2024-03-25T19:49:45.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.320+0000 7f13d7fff640 1 -- 172.21.15.110:0/2016794953 learned_addr learned my addr 172.21.15.110:0/2016794953 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:45.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.321+0000 7f13d57fa640 10 client.?.objecter ms_handle_connect 0x7f13d80aead0 2024-03-25T19:49:45.320 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.321+0000 7f13d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:45.326 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.326+0000 7f13d57fa640 10 client.?.objecter ms_handle_connect 0x7f13d806f3d0 2024-03-25T19:49:45.326 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.326+0000 7f13d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:45.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.334+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929065489 0 0) 0x7f13d813eef0 con 0x7f13d806f3d0 2024-03-25T19:49:45.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.334+0000 7f13d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:45.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.334+0000 7f13d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:45.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.334+0000 7f13d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:45.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.334+0000 7f13d57fa640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:49:45.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.334+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc004eb0 con 0x7f13d806f3d0 2024-03-25T19:49:45.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.337+0000 7f13d57fa640 10 client.?.objecter ms_handle_connect 0x7f13d8135c10 2024-03-25T19:49:45.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.337+0000 7f13d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:45.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.340+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392569814 0 0) 0x7f13d813cfe0 con 0x7f13d80aead0 2024-03-25T19:49:45.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.340+0000 7f13d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:45.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.340+0000 7f13d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:45.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.340+0000 7f13d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:45.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.340+0000 7f13d57fa640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:49:45.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.340+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d813eef0 con 0x7f13d80aead0 2024-03-25T19:49:45.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.342+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687525017 0 0) 0x7f13d8136410 con 0x7f13d8135c10 2024-03-25T19:49:45.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.342+0000 7f13d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:45.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.342+0000 7f13d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:45.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.342+0000 7f13d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:45.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.342+0000 7f13d57fa640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:49:45.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.342+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d813cfe0 con 0x7f13d8135c10 2024-03-25T19:49:45.347 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 477040048 0 0) 0x7f13d813eef0 con 0x7f13d80aead0 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d8136410 con 0x7f13d80aead0 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f13c4001850 con 0x7f13d80aead0 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 10 monclient(hunting): dump: 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:45.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:45.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1881823221 0 0) 0x7f13bc004eb0 con 0x7f13d806f3d0 2024-03-25T19:49:45.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d813eef0 con 0x7f13d806f3d0 2024-03-25T19:49:45.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f13cc001bc0 con 0x7f13d806f3d0 2024-03-25T19:49:45.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:45.349 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.348+0000 7f13d57fa640 10 monclient(hunting): dump: 2024-03-25T19:49:45.349 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:45.349 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:45.350 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.350 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.350 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.350 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:45.350 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:45.350 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:45.350 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:45.350 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.351+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3032474239 0 0) 0x7f13d813cfe0 con 0x7f13d8135c10 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.351+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc004eb0 con 0x7f13d8135c10 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.351+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f13d0001cb0 con 0x7f13d8135c10 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.351+0000 7f13d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.351+0000 7f13d57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.351+0000 7f13d57fa640 10 monclient(hunting): dump: 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.351 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:45.352 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:45.352 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:45.352 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:45.352 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:45.355 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1247898237 0 0) 0x7f13d813eef0 con 0x7f13d806f3d0 2024-03-25T19:49:45.355 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 >> v1:172.21.15.178:6789/0 conn(0x7f13d8135c10 legacy=0x7f13d81e30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.355 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 >> v1:172.21.15.110:6789/0 conn(0x7f13d80aead0 legacy=0x7f13d81da0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.355 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:49:45.355 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 1 monclient: found mon.c 2024-03-25T19:49:45.355 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:45.355 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d813d990 con 0x7f13d806f3d0 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 10 monclient: _finish_auth 0 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 5 monclient: authenticate success, global_id 4655 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 10 monclient: _renew_subs 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d813d520 con 0x7f13d806f3d0 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 10 monclient: _renew_subs 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d81dbb50 con 0x7f13d806f3d0 2024-03-25T19:49:45.356 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.355+0000 7f13decc1640 1 librados: init done 2024-03-25T19:49:45.357 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.356+0000 7f13decc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:45.357 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.356+0000 7f13decc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:45.357 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.356+0000 7f13decc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:45.357 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.356+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d813ce40 con 0x7f13d806f3d0 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.648+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13cc0038a0 con 0x7f13d806f3d0 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.648+0000 7f13d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.648+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f13cc001250 con 0x7f13d806f3d0 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.648+0000 7f13d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.648+0000 7f13d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.648+0000 7f13d57fa640 10 monclient: dump: 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.648 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:45.649 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.649 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:45.649 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:45.649 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:45.649 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:45.649 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:45.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.648+0000 7f13d6ffd640 4 set_mon_vals no callback set 2024-03-25T19:49:45.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.649+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f13cc0198c0 con 0x7f13d806f3d0 2024-03-25T19:49:45.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.650+0000 7f13d7fff640 10 monclient: get_auth_request con 0x7f13bc06e110 auth_method 0 2024-03-25T19:49:45.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.650+0000 7f13d57fa640 10 client.4655.objecter ms_handle_connect 0x7f13bc06e110 2024-03-25T19:49:45.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.655+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f13cc053560 con 0x7f13d806f3d0 2024-03-25T19:49:45.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.656+0000 7f13d57fa640 10 client.4655.objecter ms_dispatch 0x7f13d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:45.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.656+0000 7f13d57fa640 3 client.4655.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:45.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.656+0000 7f13d57fa640 3 client.4655.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:45.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.656+0000 7f13d57fa640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-25T19:49:45.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.656+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13cc04eea0 con 0x7f13d806f3d0 2024-03-25T19:49:45.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.656+0000 7f13d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:45.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.656+0000 7f13d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:45.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.899+0000 7f13decc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:45.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.899+0000 7f13decc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:45.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.899+0000 7f13decc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:49:45.899 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.899+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13d80627b0 con 0x7f13d806f3d0 2024-03-25T19:49:45.919 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.920+0000 7f13d57fa640 1 -- 172.21.15.110:0/2016794953 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f13cc050fa0 con 0x7f13d806f3d0 2024-03-25T19:49:45.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.920+0000 7f13d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:45.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.920+0000 7f13d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:45.920 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:49:45.920 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:45.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 10 librados: watch_flush enter 2024-03-25T19:49:45.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 10 librados: watch_flush exit 2024-03-25T19:49:45.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-25T19:49:45.922 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 10 client.4655.objecter successfully canceled tick 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 >> v1:172.21.15.178:6800/1132739058 conn(0x7f13bc06e110 legacy=0x7f13bc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 10 monclient: shutdown 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 >> v1:172.21.15.110:6790/0 conn(0x7f13d806f3d0 legacy=0x7f13d81db260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 shutdown_connections 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 >> 172.21.15.110:0/2016794953 conn(0x7f13d80b3e60 msgr2=0x7f13d806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 shutdown_connections 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 1 -- 172.21.15.110:0/2016794953 wait complete. 2024-03-25T19:49:45.923 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:45.922+0000 7f13decc1640 1 librados: shutdown 2024-03-25T19:49:45.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:52.741 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.852+0000 7fda1dc0f640 10 monclient: get_monmap_and_config 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.852+0000 7fda1dc0f640 10 monclient: build_initial_monmap 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.852+0000 7fda1dc0f640 10 monclient: monmap: 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:49:52.853503+0000 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:49:52.853503+0000 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:49:52.852 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:49:52.853 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:52.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.853+0000 7fda1dc0f640 1 Processor -- start 2024-03-25T19:49:52.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.853+0000 7fda1dc0f640 1 -- start start 2024-03-25T19:49:52.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.853+0000 7fda1dc0f640 10 monclient: init 2024-03-25T19:49:52.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:52.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:52.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 10 monclient(hunting): picked mon.noname-a con 0x7fda1806f380 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:52.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 10 monclient(hunting): picked mon.noname-c con 0x7fda1806f760 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:52.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 10 monclient(hunting): picked mon.noname-b con 0x7fda18072670 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:52.855 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda180718b0 con 0x7fda1806f380 2024-03-25T19:49:52.855 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1813efc0 con 0x7fda1806f760 2024-03-25T19:49:52.855 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda18070120 con 0x7fda18072670 2024-03-25T19:49:52.855 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:52.855 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.854+0000 7fda1dc0f640 10 monclient(hunting): authenticate will time out at 4763.995295s 2024-03-25T19:49:52.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.858+0000 7fda17fff640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7fda18072670 0x7fda1814a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:34292/0 (socket says 172.21.15.110:34292) 2024-03-25T19:49:52.858 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.858+0000 7fda17fff640 1 -- 172.21.15.110:0/61403085 learned_addr learned my addr 172.21.15.110:0/61403085 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:52.917 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.917+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952718715 0 0) 0x7fda1813efc0 con 0x7fda1806f760 2024-03-25T19:49:52.917 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.918+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.917 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.918+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.918+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.918+0000 7fda167fc640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:49:52.918 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.918+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda00005730 con 0x7fda1806f760 2024-03-25T19:49:52.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.920+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671051983 0 0) 0x7fda180718b0 con 0x7fda1806f380 2024-03-25T19:49:52.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.920+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.920+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.920 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.920+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.920+0000 7fda167fc640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:49:52.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.920+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda1813efc0 con 0x7fda1806f380 2024-03-25T19:49:52.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.921+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519137855 0 0) 0x7fda18070120 con 0x7fda18072670 2024-03-25T19:49:52.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.921+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.921+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.921+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.921+0000 7fda167fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:49:52.921 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.921+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda180718b0 con 0x7fda18072670 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 963999006 0 0) 0x7fda180718b0 con 0x7fda18072670 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 >> v1:172.21.15.178:6789/0 conn(0x7fda1806f760 legacy=0x7fda18071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 >> v1:172.21.15.110:6789/0 conn(0x7fda1806f380 legacy=0x7fda1813f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 1 monclient: found mon.noname-b 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18070610 con 0x7fda18072670 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda1dc0f640 5 monclient: authenticate success, global_id 4658 2024-03-25T19:49:52.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fda10001930 con 0x7fda18072670 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda1dc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda167fc640 10 monclient: dump: 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:52.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.937+0000 7fda1dc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:52.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda10001a90 con 0x7fda18072670 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda167fc640 1 -- 172.21.15.110:0/61403085 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fda10001ee0 con 0x7fda18072670 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda167fc640 10 monclient: dump: 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:52.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda1dc0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:52.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda1dc0f640 4 set_mon_vals no callback set 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda1dc0f640 10 monclient: shutdown 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda1dc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda1dc0f640 1 -- 172.21.15.110:0/61403085 >> v1:172.21.15.110:6790/0 conn(0x7fda18072670 legacy=0x7fda1814a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda1dc0f640 1 -- 172.21.15.110:0/61403085 shutdown_connections 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.952+0000 7fda1dc0f640 1 -- 172.21.15.110:0/61403085 >> 172.21.15.110:0/61403085 conn(0x7fda1806c830 msgr2=0x7fda180b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.953+0000 7fda1dc0f640 1 -- 172.21.15.110:0/61403085 shutdown_connections 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.953+0000 7fda1dc0f640 1 -- 172.21.15.110:0/61403085 wait complete. 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.953+0000 7fda1dc0f640 10 monclient: build_initial_monmap 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.953+0000 7fda1dc0f640 10 monclient: monmap: 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:49:52.954 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.954+0000 7fda1dc0f640 1 librados: starting msgr at 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.954+0000 7fda1dc0f640 1 librados: starting objecter 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.954+0000 7fda1dc0f640 1 Processor -- start 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.954+0000 7fda1dc0f640 1 -- start start 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.954+0000 7fda1dc0f640 1 librados: setting wanted keys 2024-03-25T19:49:52.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.954+0000 7fda1dc0f640 1 librados: calling monclient init 2024-03-25T19:49:52.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.954+0000 7fda1dc0f640 10 monclient: init 2024-03-25T19:49:52.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:52.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:52.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 10 monclient(hunting): picked mon.noname-b con 0x7fda1806f380 addr v1:172.21.15.178:6789/0 2024-03-25T19:49:52.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 10 monclient(hunting): picked mon.noname-a con 0x7fda1806f760 addr v1:172.21.15.110:6789/0 2024-03-25T19:49:52.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 10 monclient(hunting): picked mon.noname-c con 0x7fda18072670 addr v1:172.21.15.110:6790/0 2024-03-25T19:49:52.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda180aa940 con 0x7fda1806f760 2024-03-25T19:49:52.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda181368d0 con 0x7fda1806f380 2024-03-25T19:49:52.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda180aabb0 con 0x7fda18072670 2024-03-25T19:49:52.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:52.957 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.955+0000 7fda1dc0f640 10 monclient(hunting): authenticate will time out at 4764.096219s 2024-03-25T19:49:52.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.958+0000 7fda177fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fda1806f380 0x7fda181dfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:43848/0 (socket says 172.21.15.110:43848) 2024-03-25T19:49:52.958 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.958+0000 7fda177fe640 1 -- 172.21.15.110:0/626124805 learned_addr learned my addr 172.21.15.110:0/626124805 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:49:52.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.959+0000 7fd9fffff640 10 client.?.objecter ms_handle_connect 0x7fda1806f380 2024-03-25T19:49:52.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.959+0000 7fd9fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:52.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.964+0000 7fd9fffff640 10 client.?.objecter ms_handle_connect 0x7fda18072670 2024-03-25T19:49:52.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.964+0000 7fd9fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:52.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.967+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263603021 0 0) 0x7fda181368d0 con 0x7fda1806f380 2024-03-25T19:49:52.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.967+0000 7fd9fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.967+0000 7fd9fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.967+0000 7fd9fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.967+0000 7fd9fffff640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:49:52.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.967+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda04004eb0 con 0x7fda1806f380 2024-03-25T19:49:52.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.970+0000 7fd9fffff640 10 client.?.objecter ms_handle_connect 0x7fda1806f760 2024-03-25T19:49:52.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.970+0000 7fd9fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.976+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489095508 0 0) 0x7fda180aabb0 con 0x7fda18072670 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.976+0000 7fd9fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.976+0000 7fd9fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.976+0000 7fd9fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.976+0000 7fd9fffff640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.976+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda181368d0 con 0x7fda18072670 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.991+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 288088038 0 0) 0x7fda04004eb0 con 0x7fda1806f380 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.991+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda180aabb0 con 0x7fda1806f380 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.991+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fda08001a10 con 0x7fda1806f380 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.991+0000 7fd9fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.991+0000 7fd9fffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.991+0000 7fd9fffff640 10 monclient(hunting): dump: 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:52.991 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:52.992 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:52.992 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:52.992 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:52.992 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:52.992 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:52.992 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:52.992 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:52.992 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.991+0000 7fd9fffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:52.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.994+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150982775 0 0) 0x7fda180aa940 con 0x7fda1806f760 2024-03-25T19:49:52.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.994+0000 7fd9fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.994+0000 7fd9fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.994+0000 7fd9fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.994+0000 7fd9fffff640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:49:52.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:52.994+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda04004eb0 con 0x7fda1806f760 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3489415776 0 0) 0x7fda180aabb0 con 0x7fda1806f380 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 >> v1:172.21.15.110:6790/0 conn(0x7fda18072670 legacy=0x7fda181e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 >> v1:172.21.15.110:6789/0 conn(0x7fda1806f760 legacy=0x7fda181e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 1 monclient: found mon.b 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda180a97e0 con 0x7fda1806f380 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 10 monclient: _finish_auth 0 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fd9fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:53.010 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 10 monclient: _renew_subs 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda18136560 con 0x7fda1806f380 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 10 monclient: _renew_subs 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda18143120 con 0x7fda1806f380 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.010+0000 7fda1dc0f640 1 librados: init done 2024-03-25T19:49:53.011 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.011+0000 7fda1dc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:53.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.011+0000 7fda1dc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:53.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.011+0000 7fda1dc0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:53.012 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.011+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda18070b00 con 0x7fda1806f380 2024-03-25T19:49:53.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.277+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda08001c50 con 0x7fda1806f380 2024-03-25T19:49:53.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.277+0000 7fd9fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.277+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fda08003a60 con 0x7fda1806f380 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.277+0000 7fd9fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.277+0000 7fd9fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.277+0000 7fd9fffff640 10 monclient: dump: 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:53.277 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:49:53.278 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:49:53.278 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:49:53.278 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:49:53.278 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:49:53.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.277+0000 7fda15ffb640 4 set_mon_vals no callback set 2024-03-25T19:49:53.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.278+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fda08003c80 con 0x7fda1806f380 2024-03-25T19:49:53.278 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.278+0000 7fda16ffd640 10 monclient: get_auth_request con 0x7fda0406de00 auth_method 0 2024-03-25T19:49:53.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.279+0000 7fd9fffff640 10 client.4657.objecter ms_handle_connect 0x7fda0406de00 2024-03-25T19:49:53.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.285+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fda08054880 con 0x7fda1806f380 2024-03-25T19:49:53.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.285+0000 7fd9fffff640 10 client.4657.objecter ms_dispatch 0x7fda18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:53.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.285+0000 7fd9fffff640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:53.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.285+0000 7fd9fffff640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:53.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.285+0000 7fd9fffff640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T19:49:53.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.285+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda08050390 con 0x7fda1806f380 2024-03-25T19:49:53.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.285+0000 7fd9fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:53.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.285+0000 7fd9fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:53.531 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.531+0000 7fda1dc0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:53.531 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.531+0000 7fda1dc0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:53.531 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.531+0000 7fda1dc0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:49:53.531 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.531+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda181366f0 con 0x7fda1806f380 2024-03-25T19:49:53.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.561+0000 7fd9fffff640 1 -- 172.21.15.110:0/626124805 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fda080522c0 con 0x7fda1806f380 2024-03-25T19:49:53.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.561+0000 7fd9fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:53.561 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.561+0000 7fd9fffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:53.561 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:49:53.562 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:53.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 10 librados: watch_flush enter 2024-03-25T19:49:53.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 10 librados: watch_flush exit 2024-03-25T19:49:53.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T19:49:53.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 10 client.4657.objecter successfully canceled tick 2024-03-25T19:49:53.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 >> v1:172.21.15.178:6800/1132739058 conn(0x7fda0406de00 legacy=0x7fda040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:53.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 10 monclient: shutdown 2024-03-25T19:49:53.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:53.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 >> v1:172.21.15.178:6789/0 conn(0x7fda1806f380 legacy=0x7fda181dfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:53.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 shutdown_connections 2024-03-25T19:49:53.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 >> 172.21.15.110:0/626124805 conn(0x7fda1806c830 msgr2=0x7fda18074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:53.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 shutdown_connections 2024-03-25T19:49:53.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 1 -- 172.21.15.110:0/626124805 wait complete. 2024-03-25T19:49:53.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:49:53.564+0000 7fda1dc0f640 1 librados: shutdown 2024-03-25T19:49:53.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:00.382 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.499+0000 7f355aaa1640 10 monclient: get_monmap_and_config 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.499+0000 7f355aaa1640 10 monclient: build_initial_monmap 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.499+0000 7f355aaa1640 10 monclient: monmap: 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:50:00.500236+0000 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:50:00.500236+0000 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:50:00.499 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:50:00.500 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.500+0000 7f355aaa1640 1 Processor -- start 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.500+0000 7f355aaa1640 1 -- start start 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.500+0000 7f355aaa1640 10 monclient: init 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.500+0000 7f355aaa1640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.500+0000 7f355aaa1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.500+0000 7f355aaa1640 10 monclient(hunting): picked mon.noname-b con 0x7f35540a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.501+0000 7f355aaa1640 10 monclient(hunting): picked mon.noname-a con 0x7f355413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.501+0000 7f355aaa1640 10 monclient(hunting): picked mon.noname-c con 0x7f35541430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.501+0000 7f355aaa1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3554142520 con 0x7f355413f710 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.501+0000 7f355aaa1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35540b28a0 con 0x7f35541430c0 2024-03-25T19:50:00.501 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.501+0000 7f355aaa1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35540b2a30 con 0x7f35540a98f0 2024-03-25T19:50:00.502 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.501+0000 7f355aaa1640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:00.502 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.501+0000 7f355aaa1640 10 monclient(hunting): authenticate will time out at 4771.641981s 2024-03-25T19:50:00.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.509+0000 7f3553fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f355413f710 0x7f35541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56376/0 (socket says 172.21.15.110:56376) 2024-03-25T19:50:00.509 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.509+0000 7f3553fff640 1 -- 172.21.15.110:0/662702799 learned_addr learned my addr 172.21.15.110:0/662702799 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:00.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.518+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453836654 0 0) 0x7f3554142520 con 0x7f355413f710 2024-03-25T19:50:00.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.518+0000 7f35537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.518+0000 7f35537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.518+0000 7f35537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.518+0000 7f35537fe640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:50:00.518 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.518+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3540003520 con 0x7f355413f710 2024-03-25T19:50:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.520+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849431113 0 0) 0x7f35540b2a30 con 0x7f35540a98f0 2024-03-25T19:50:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.520+0000 7f35537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.520+0000 7f35537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.520+0000 7f35537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.520+0000 7f35537fe640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:50:00.520 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.520+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3554142520 con 0x7f35540a98f0 2024-03-25T19:50:00.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.522+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195093717 0 0) 0x7f35540b28a0 con 0x7f35541430c0 2024-03-25T19:50:00.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.522+0000 7f35537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.522+0000 7f35537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.521 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.522+0000 7f35537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.522+0000 7f35537fe640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:50:00.522 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.522+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35540b2a30 con 0x7f35541430c0 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244204605 0 0) 0x7f35540b2a30 con 0x7f35541430c0 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 >> v1:172.21.15.110:6790/0 conn(0x7f35540a98f0 legacy=0x7f35540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 >> v1:172.21.15.110:6789/0 conn(0x7f355413f710 legacy=0x7f35541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 1 monclient: found mon.noname-c 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35540b44c0 con 0x7f35541430c0 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:00.532 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f354c001930 con 0x7f35541430c0 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f35537fe640 10 monclient: dump: 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f355aaa1640 5 monclient: authenticate success, global_id 4660 2024-03-25T19:50:00.533 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.532+0000 7f355aaa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:00.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.539+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f354c001a90 con 0x7f35541430c0 2024-03-25T19:50:00.539 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.539+0000 7f35537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.539+0000 7f35537fe640 1 -- 172.21.15.110:0/662702799 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f354c001ee0 con 0x7f35541430c0 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.539+0000 7f355aaa1640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.539+0000 7f355aaa1640 4 set_mon_vals no callback set 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.539+0000 7f35537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.539+0000 7f35537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.539+0000 7f35537fe640 10 monclient: dump: 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:00.540 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:00.541 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:00.541 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:00.544 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.540+0000 7f355aaa1640 10 monclient: shutdown 2024-03-25T19:50:00.544 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.540+0000 7f355aaa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:00.544 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.540+0000 7f355aaa1640 1 -- 172.21.15.110:0/662702799 >> v1:172.21.15.178:6789/0 conn(0x7f35541430c0 legacy=0x7f355414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.544 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.540+0000 7f355aaa1640 1 -- 172.21.15.110:0/662702799 shutdown_connections 2024-03-25T19:50:00.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.540+0000 7f355aaa1640 1 -- 172.21.15.110:0/662702799 >> 172.21.15.110:0/662702799 conn(0x7f35540b7ea0 msgr2=0x7f35540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:00.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.540+0000 7f355aaa1640 1 -- 172.21.15.110:0/662702799 shutdown_connections 2024-03-25T19:50:00.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.540+0000 7f355aaa1640 1 -- 172.21.15.110:0/662702799 wait complete. 2024-03-25T19:50:00.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 10 monclient: build_initial_monmap 2024-03-25T19:50:00.545 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 10 monclient: monmap: 2024-03-25T19:50:00.545 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:00.545 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:00.546 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:50:00.546 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:50:00.546 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:00.546 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:50:00.546 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:00.547 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:50:00.547 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:50:00.547 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:00.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 1 librados: starting msgr at 2024-03-25T19:50:00.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 1 librados: starting objecter 2024-03-25T19:50:00.547 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 1 Processor -- start 2024-03-25T19:50:00.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 1 -- start start 2024-03-25T19:50:00.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 1 librados: setting wanted keys 2024-03-25T19:50:00.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 1 librados: calling monclient init 2024-03-25T19:50:00.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.541+0000 7f355aaa1640 10 monclient: init 2024-03-25T19:50:00.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:00.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:00.548 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 10 monclient(hunting): picked mon.noname-a con 0x7f35540a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:00.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 10 monclient(hunting): picked mon.noname-c con 0x7f355413f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:00.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 10 monclient(hunting): picked mon.noname-b con 0x7f35541430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:00.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3554138b50 con 0x7f35540a98f0 2024-03-25T19:50:00.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3554139950 con 0x7f35541430c0 2024-03-25T19:50:00.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3554138070 con 0x7f355413f710 2024-03-25T19:50:00.549 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:00.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.542+0000 7f355aaa1640 10 monclient(hunting): authenticate will time out at 4771.683681s 2024-03-25T19:50:00.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.548+0000 7f355883d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f35540a98f0 0x7f3554137840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56416/0 (socket says 172.21.15.110:56416) 2024-03-25T19:50:00.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.548+0000 7f355883d640 1 -- 172.21.15.110:0/1387890334 learned_addr learned my addr 172.21.15.110:0/1387890334 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:00.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.549+0000 7f35517fa640 10 client.?.objecter ms_handle_connect 0x7f35541430c0 2024-03-25T19:50:00.550 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.549+0000 7f35517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:00.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.558+0000 7f35517fa640 10 client.?.objecter ms_handle_connect 0x7f355413f710 2024-03-25T19:50:00.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.558+0000 7f35517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:00.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.559+0000 7f35517fa640 10 client.?.objecter ms_handle_connect 0x7f35540a98f0 2024-03-25T19:50:00.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.559+0000 7f35517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:00.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.569+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714803999 0 0) 0x7f3554139950 con 0x7f35541430c0 2024-03-25T19:50:00.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.569+0000 7f35517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.569+0000 7f35517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.569+0000 7f35517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.569+0000 7f35517fa640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:50:00.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.569+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3534004eb0 con 0x7f35541430c0 2024-03-25T19:50:00.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.573+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779038619 0 0) 0x7f3554138b50 con 0x7f35540a98f0 2024-03-25T19:50:00.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.573+0000 7f35517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.573+0000 7f35517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.573+0000 7f35517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.573+0000 7f35517fa640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:50:00.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.573+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3554139950 con 0x7f35540a98f0 2024-03-25T19:50:00.580 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.580+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588038506 0 0) 0x7f3554138070 con 0x7f355413f710 2024-03-25T19:50:00.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.580+0000 7f35517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.580+0000 7f35517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.580+0000 7f35517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.580+0000 7f35517fa640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:50:00.581 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.581+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3554138b50 con 0x7f355413f710 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.585+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3423158439 0 0) 0x7f3534004eb0 con 0x7f35541430c0 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.585+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3554138070 con 0x7f35541430c0 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.585+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f354c001de0 con 0x7f35541430c0 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.585+0000 7f35517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.585+0000 7f35517fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.585+0000 7f35517fa640 10 monclient(hunting): dump: 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.585 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.586 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:00.586 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:00.586 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:00.586 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:00.586 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:00.586 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.585+0000 7f35517fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:00.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.597+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2656649249 0 0) 0x7f3554139950 con 0x7f35540a98f0 2024-03-25T19:50:00.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.597+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3534004eb0 con 0x7f35540a98f0 2024-03-25T19:50:00.597 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.597+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f35440019f0 con 0x7f35540a98f0 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.597+0000 7f35517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.597+0000 7f35517fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.597+0000 7f35517fa640 10 monclient(hunting): dump: 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:00.598 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:00.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.603+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839435442 0 0) 0x7f3554138b50 con 0x7f355413f710 2024-03-25T19:50:00.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.603+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3554139950 con 0x7f355413f710 2024-03-25T19:50:00.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.603+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f353c0018d0 con 0x7f355413f710 2024-03-25T19:50:00.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.603+0000 7f35517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.603+0000 7f35517fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:00.603 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.603+0000 7f35517fa640 10 monclient(hunting): dump: 2024-03-25T19:50:00.603 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:00.604 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 286513709 0 0) 0x7f3554138070 con 0x7f35541430c0 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 >> v1:172.21.15.110:6790/0 conn(0x7f355413f710 legacy=0x7f35541da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 >> v1:172.21.15.110:6789/0 conn(0x7f35540a98f0 legacy=0x7f3554137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 10 monclient: _finish_hunting 0 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 1 monclient: found mon.b 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3554138200 con 0x7f35541430c0 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 10 monclient: _finish_auth 0 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 10 monclient: _check_auth_tickets 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.612+0000 7f35517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:50:00.613 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 10 monclient: _renew_subs 2024-03-25T19:50:00.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:00.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35541383f0 con 0x7f35541430c0 2024-03-25T19:50:00.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:00.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 10 monclient: _renew_subs 2024-03-25T19:50:00.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:00.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35541db170 con 0x7f35541430c0 2024-03-25T19:50:00.614 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 1 librados: init done 2024-03-25T19:50:00.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:00.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:00.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:00.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.613+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3554074cc0 con 0x7f35541430c0 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.905+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f354c002020 con 0x7f35541430c0 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.905+0000 7f35517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.905+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f354c003dc0 con 0x7f35541430c0 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.905+0000 7f35517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.905+0000 7f35517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.905+0000 7f35517fa640 10 monclient: dump: 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:00.905 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.905+0000 7f3552ffd640 4 set_mon_vals no callback set 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.906+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f354c019d90 con 0x7f35541430c0 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.906+0000 7f355883d640 10 monclient: get_auth_request con 0x7f353406e0f0 auth_method 0 2024-03-25T19:50:00.906 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.907+0000 7f35517fa640 10 client.4663.objecter ms_handle_connect 0x7f353406e0f0 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.912+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f354c0538d0 con 0x7f35541430c0 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.912+0000 7f35517fa640 10 client.4663.objecter ms_dispatch 0x7f3554000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.912+0000 7f35517fa640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.912+0000 7f35517fa640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.912+0000 7f35517fa640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.913+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f354c04f3e0 con 0x7f35541430c0 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.913+0000 7f35517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:00.913+0000 7f35517fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:01.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.162+0000 7f355aaa1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:01.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.162+0000 7f355aaa1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:01.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.162+0000 7f355aaa1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:01.162 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.162+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35540627b0 con 0x7f35541430c0 2024-03-25T19:50:01.197 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.198+0000 7f35517fa640 1 -- 172.21.15.110:0/1387890334 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f354c051310 con 0x7f35541430c0 2024-03-25T19:50:01.197 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.198+0000 7f35517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:01.198 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.198+0000 7f35517fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:01.198 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:50:01.198 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:01.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 10 librados: watch_flush enter 2024-03-25T19:50:01.200 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 10 librados: watch_flush exit 2024-03-25T19:50:01.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T19:50:01.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 10 client.4663.objecter successfully canceled tick 2024-03-25T19:50:01.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 >> v1:172.21.15.178:6800/1132739058 conn(0x7f353406e0f0 legacy=0x7f35340705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:01.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 10 monclient: shutdown 2024-03-25T19:50:01.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:01.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 >> v1:172.21.15.178:6789/0 conn(0x7f35541430c0 legacy=0x7f35541e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:01.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 shutdown_connections 2024-03-25T19:50:01.201 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 >> 172.21.15.110:0/1387890334 conn(0x7f35540b7ea0 msgr2=0x7f3554074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:01.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 shutdown_connections 2024-03-25T19:50:01.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 1 -- 172.21.15.110:0/1387890334 wait complete. 2024-03-25T19:50:01.202 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:01.200+0000 7f355aaa1640 1 librados: shutdown 2024-03-25T19:50:01.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:03.858 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:11.142 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.861 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:11.146 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.904 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:11.188 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.917 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:11.171 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.922 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:11.206 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.927 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:11.182 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.933 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:11.217 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.950 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:11.204 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.961 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:11.245 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.018 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.140+0000 7fa9036b5640 10 monclient: get_monmap_and_config 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.140+0000 7fa9036b5640 10 monclient: build_initial_monmap 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.140+0000 7fa9036b5640 10 monclient: monmap: 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:50:08.141349+0000 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:50:08.141349+0000 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:08.140 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:50:08.141 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:50:08.141 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.141+0000 7fa9036b5640 1 Processor -- start 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.141+0000 7fa9036b5640 1 -- start start 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.141+0000 7fa9036b5640 10 monclient: init 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 10 monclient(hunting): picked mon.noname-a con 0x7fa8fc0b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 10 monclient(hunting): picked mon.noname-b con 0x7fa8fc0b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 10 monclient(hunting): picked mon.noname-c con 0x7fa8fc14be90 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8fc0af940 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8fc0b0a70 con 0x7fa8fc14be90 2024-03-25T19:50:08.142 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8fc0b0d50 con 0x7fa8fc0b1730 2024-03-25T19:50:08.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:08.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.142+0000 7fa9036b5640 10 monclient(hunting): authenticate will time out at 4779.283099s 2024-03-25T19:50:08.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.143+0000 7fa901c52640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fa8fc14be90 0x7fa8fc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:45014/0 (socket says 172.21.15.110:45014) 2024-03-25T19:50:08.143 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.143+0000 7fa901c52640 1 -- 172.21.15.110:0/3364285119 learned_addr learned my addr 172.21.15.110:0/3364285119 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:08.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.212+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528811523 0 0) 0x7fa8fc0af940 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.212+0000 7fa8f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:08.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.212+0000 7fa8f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:08.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.212+0000 7fa8f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:08.212 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.212+0000 7fa8f3fff640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:50:08.213 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.212+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e8003520 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.225+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2801984046 0 0) 0x7fa8e8003520 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 >> v1:172.21.15.110:6790/0 conn(0x7fa8fc0b1730 legacy=0x7fa8fc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 >> v1:172.21.15.178:6789/0 conn(0x7fa8fc14be90 legacy=0x7fa8fc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:08.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 1 monclient: found mon.noname-a 2024-03-25T19:50:08.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:08.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:50:08.226 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc0b1030 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 10 monclient: _finish_auth 0 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa8f8001690 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa9036b5640 5 monclient: authenticate success, global_id 4659 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa9036b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa8f3fff640 10 monclient: dump: 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:08.227 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:08.228 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:08.228 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:08.228 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:08.228 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.226+0000 7fa9036b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:08.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.236+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8f80017f0 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.236+0000 7fa8f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:08.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.236+0000 7fa8f3fff640 1 -- 172.21.15.110:0/3364285119 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa8f8001c00 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.236 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.236+0000 7fa9036b5640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.236+0000 7fa9036b5640 4 set_mon_vals no callback set 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.236+0000 7fa8f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.236+0000 7fa8f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.236+0000 7fa8f3fff640 10 monclient: dump: 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:08.237 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:08.238 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:08.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.237+0000 7fa9036b5640 10 monclient: shutdown 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.237+0000 7fa9036b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.237+0000 7fa9036b5640 1 -- 172.21.15.110:0/3364285119 >> v1:172.21.15.110:6789/0 conn(0x7fa8fc0b4d60 legacy=0x7fa8fc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.237+0000 7fa9036b5640 1 -- 172.21.15.110:0/3364285119 shutdown_connections 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.237+0000 7fa9036b5640 1 -- 172.21.15.110:0/3364285119 >> 172.21.15.110:0/3364285119 conn(0x7fa8fc0617a0 msgr2=0x7fa8fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.237+0000 7fa9036b5640 1 -- 172.21.15.110:0/3364285119 shutdown_connections 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.237+0000 7fa9036b5640 1 -- 172.21.15.110:0/3364285119 wait complete. 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.238+0000 7fa9036b5640 10 monclient: build_initial_monmap 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.238+0000 7fa9036b5640 10 monclient: monmap: 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:08.240 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.239+0000 7fa9036b5640 1 librados: starting msgr at 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.239+0000 7fa9036b5640 1 librados: starting objecter 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.239+0000 7fa9036b5640 1 Processor -- start 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.239+0000 7fa9036b5640 1 -- start start 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.239+0000 7fa9036b5640 1 librados: setting wanted keys 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.239+0000 7fa9036b5640 1 librados: calling monclient init 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.239+0000 7fa9036b5640 10 monclient: init 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 10 monclient(hunting): picked mon.noname-a con 0x7fa8fc0b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 10 monclient(hunting): picked mon.noname-c con 0x7fa8fc0b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 10 monclient(hunting): picked mon.noname-b con 0x7fa8fc14be90 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:08.241 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8fc1d7d80 con 0x7fa8fc0b1730 2024-03-25T19:50:08.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8fc1d78a0 con 0x7fa8fc14be90 2024-03-25T19:50:08.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8fc1d7a30 con 0x7fa8fc0b4d60 2024-03-25T19:50:08.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:08.242 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.240+0000 7fa9036b5640 10 monclient(hunting): authenticate will time out at 4779.381585s 2024-03-25T19:50:08.250 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.251+0000 7fa901c52640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fa8fc14be90 0x7fa8fc1369b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:45044/0 (socket says 172.21.15.110:45044) 2024-03-25T19:50:08.250 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.251+0000 7fa901c52640 1 -- 172.21.15.110:0/18711993 learned_addr learned my addr 172.21.15.110:0/18711993 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:08.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.252+0000 7fa8f1ffb640 10 client.?.objecter ms_handle_connect 0x7fa8fc14be90 2024-03-25T19:50:08.252 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.252+0000 7fa8f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:08.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.258+0000 7fa8f1ffb640 10 client.?.objecter ms_handle_connect 0x7fa8fc0b1730 2024-03-25T19:50:08.257 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.258+0000 7fa8f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:08.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.262+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124244683 0 0) 0x7fa8fc1d78a0 con 0x7fa8fc14be90 2024-03-25T19:50:08.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.262+0000 7fa8f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:08.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.262+0000 7fa8f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:08.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.262+0000 7fa8f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:08.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.262+0000 7fa8f1ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:50:08.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.262+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e0004eb0 con 0x7fa8fc14be90 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.269+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1509594860 0 0) 0x7fa8e0004eb0 con 0x7fa8fc14be90 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.269+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8fc1d78a0 con 0x7fa8fc14be90 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.269+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa8f4001b10 con 0x7fa8fc14be90 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.269+0000 7fa8f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.269+0000 7fa8f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.269+0000 7fa8f1ffb640 10 monclient(hunting): dump: 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:08.269 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:08.270 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:08.270 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:08.270 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:08.270 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.269+0000 7fa8f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:08.274 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.275+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287352138 0 0) 0x7fa8fc1d7d80 con 0x7fa8fc0b1730 2024-03-25T19:50:08.274 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.275+0000 7fa8f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:08.274 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.275+0000 7fa8f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:08.275 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.275+0000 7fa8f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:08.275 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.275+0000 7fa8f1ffb640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:50:08.275 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.275+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e0004eb0 con 0x7fa8fc0b1730 2024-03-25T19:50:08.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 833149421 0 0) 0x7fa8fc1d78a0 con 0x7fa8fc14be90 2024-03-25T19:50:08.289 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 >> v1:172.21.15.110:6790/0 conn(0x7fa8fc0b4d60 legacy=0x7fa8fc143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 >> v1:172.21.15.110:6789/0 conn(0x7fa8fc0b1730 legacy=0x7fa8fc1d8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 1 monclient: found mon.b 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc136200 con 0x7fa8fc14be90 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa8f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 10 monclient: _renew_subs 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8fc1364d0 con 0x7fa8fc14be90 2024-03-25T19:50:08.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:08.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 10 monclient: _renew_subs 2024-03-25T19:50:08.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:08.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8fc130040 con 0x7fa8fc14be90 2024-03-25T19:50:08.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.289+0000 7fa9036b5640 1 librados: init done 2024-03-25T19:50:08.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.290+0000 7fa9036b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:08.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.290+0000 7fa9036b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:08.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.290+0000 7fa9036b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:08.292 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.290+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc1d7bc0 con 0x7fa8fc14be90 2024-03-25T19:50:08.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.568+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8f4002080 con 0x7fa8fc14be90 2024-03-25T19:50:08.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.568+0000 7fa8f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:08.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.568+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7fa8f4004b60 con 0x7fa8fc14be90 2024-03-25T19:50:08.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.568+0000 7fa8f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.568+0000 7fa8f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.568+0000 7fa8f1ffb640 10 monclient: dump: 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:08.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.568+0000 7fa8f37fe640 4 set_mon_vals no callback set 2024-03-25T19:50:08.570 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.569+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7fa8f4004d80 con 0x7fa8fc14be90 2024-03-25T19:50:08.570 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.570+0000 7fa901451640 10 monclient: get_auth_request con 0x7fa8e006ddc0 auth_method 0 2024-03-25T19:50:08.570 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.570+0000 7fa8f1ffb640 10 client.4669.objecter ms_handle_connect 0x7fa8e006ddc0 2024-03-25T19:50:08.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.574+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7fa8f4054860 con 0x7fa8fc14be90 2024-03-25T19:50:08.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.574+0000 7fa8f1ffb640 10 client.4669.objecter ms_dispatch 0x7fa8fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:08.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.574+0000 7fa8f1ffb640 3 client.4669.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:08.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.574+0000 7fa8f1ffb640 3 client.4669.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:08.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.574+0000 7fa8f1ffb640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T19:50:08.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.574+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8f40501a0 con 0x7fa8fc14be90 2024-03-25T19:50:08.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.574+0000 7fa8f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:08.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.574+0000 7fa8f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:08.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.813+0000 7fa9036b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:08.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.813+0000 7fa9036b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:08.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.813+0000 7fa9036b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:08.813 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.813+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8fc1d7f30 con 0x7fa8fc14be90 2024-03-25T19:50:08.849 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.849+0000 7fa8f1ffb640 1 -- 172.21.15.110:0/18711993 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7fa8fc1d7f30 con 0x7fa8fc14be90 2024-03-25T19:50:08.849 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.849+0000 7fa8f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:08.849 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.849+0000 7fa8f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:08.850 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:50:08.850 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:08.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 10 librados: watch_flush enter 2024-03-25T19:50:08.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 10 librados: watch_flush exit 2024-03-25T19:50:08.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T19:50:08.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 10 client.4669.objecter successfully canceled tick 2024-03-25T19:50:08.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 >> v1:172.21.15.178:6800/1132739058 conn(0x7fa8e006ddc0 legacy=0x7fa8e0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 10 monclient: shutdown 2024-03-25T19:50:08.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:08.852 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 >> v1:172.21.15.178:6789/0 conn(0x7fa8fc14be90 legacy=0x7fa8fc1369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 shutdown_connections 2024-03-25T19:50:08.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 >> 172.21.15.110:0/18711993 conn(0x7fa8fc0617a0 msgr2=0x7fa8fc0b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:08.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 shutdown_connections 2024-03-25T19:50:08.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 1 -- 172.21.15.110:0/18711993 wait complete. 2024-03-25T19:50:08.853 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:08.852+0000 7fa9036b5640 1 librados: shutdown 2024-03-25T19:50:08.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:15.669 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:15.784 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.784+0000 7f58d9cfb640 10 monclient: get_monmap_and_config 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.784+0000 7f58d9cfb640 10 monclient: build_initial_monmap 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.784+0000 7f58d9cfb640 10 monclient: monmap: 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:50:15.786164+0000 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:50:15.786164+0000 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:50:15.785 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 1 Processor -- start 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 1 -- start start 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 10 monclient: init 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f58d40b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f58d40b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f58d414be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d40af940 con 0x7f58d414be90 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d40b0a70 con 0x7f58d40b4d60 2024-03-25T19:50:15.787 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d40b0d50 con 0x7f58d40b1730 2024-03-25T19:50:15.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:15.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.785+0000 7f58d9cfb640 10 monclient(hunting): authenticate will time out at 4786.927826s 2024-03-25T19:50:15.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.786+0000 7f58d37fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f58d40b4d60 0x7f58d413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:47632/0 (socket says 172.21.15.110:47632) 2024-03-25T19:50:15.788 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.786+0000 7f58d37fe640 1 -- 172.21.15.110:0/3825766798 learned_addr learned my addr 172.21.15.110:0/3825766798 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:15.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.854+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725797022 0 0) 0x7f58d40af940 con 0x7f58d414be90 2024-03-25T19:50:15.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.854+0000 7f58d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:15.854 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.854+0000 7f58d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:15.855 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.854+0000 7f58d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:15.855 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.854+0000 7f58d2ffd640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:50:15.855 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.854+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c0003520 con 0x7f58d414be90 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3463199949 0 0) 0x7f58c0003520 con 0x7f58d414be90 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 >> v1:172.21.15.110:6790/0 conn(0x7f58d40b1730 legacy=0x7f58d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 >> v1:172.21.15.178:6789/0 conn(0x7f58d40b4d60 legacy=0x7f58d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 1 monclient: found mon.noname-a 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d40b1030 con 0x7f58d414be90 2024-03-25T19:50:15.877 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58cc001a10 con 0x7f58d414be90 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d2ffd640 10 monclient: dump: 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:15.878 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:15.879 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:15.879 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:15.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d9cfb640 5 monclient: authenticate success, global_id 4665 2024-03-25T19:50:15.879 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.876+0000 7f58d9cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.885+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58cc001b70 con 0x7f58d414be90 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.885+0000 7f58d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.885+0000 7f58d2ffd640 1 -- 172.21.15.110:0/3825766798 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58cc001f80 con 0x7f58d414be90 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.885+0000 7f58d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.885+0000 7f58d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.885+0000 7f58d2ffd640 10 monclient: dump: 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:15.886 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:15.887 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:15.887 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:15.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.885+0000 7f58d9cfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:15.887 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.885+0000 7f58d9cfb640 4 set_mon_vals no callback set 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.886+0000 7f58d9cfb640 10 monclient: shutdown 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.886+0000 7f58d9cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.886+0000 7f58d9cfb640 1 -- 172.21.15.110:0/3825766798 >> v1:172.21.15.110:6789/0 conn(0x7f58d414be90 legacy=0x7f58d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.886+0000 7f58d9cfb640 1 -- 172.21.15.110:0/3825766798 shutdown_connections 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.886+0000 7f58d9cfb640 1 -- 172.21.15.110:0/3825766798 >> 172.21.15.110:0/3825766798 conn(0x7f58d40617a0 msgr2=0x7f58d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.886+0000 7f58d9cfb640 1 -- 172.21.15.110:0/3825766798 shutdown_connections 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.886+0000 7f58d9cfb640 1 -- 172.21.15.110:0/3825766798 wait complete. 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 10 monclient: build_initial_monmap 2024-03-25T19:50:15.888 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 10 monclient: monmap: 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 1 librados: starting msgr at 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 1 librados: starting objecter 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 1 Processor -- start 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 1 -- start start 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 1 librados: setting wanted keys 2024-03-25T19:50:15.889 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 1 librados: calling monclient init 2024-03-25T19:50:15.890 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.887+0000 7f58d9cfb640 10 monclient: init 2024-03-25T19:50:15.890 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f58d40b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f58d40b4d60 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f58d414be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d41307c0 con 0x7f58d40b4d60 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d4138f90 con 0x7f58d40b1730 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d4132ab0 con 0x7f58d414be90 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:15.891 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.888+0000 7f58d9cfb640 10 monclient(hunting): authenticate will time out at 4787.030452s 2024-03-25T19:50:15.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.902+0000 7f58cbfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f58d40b4d60 0x7f58d412f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44418/0 (socket says 172.21.15.110:44418) 2024-03-25T19:50:15.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.902+0000 7f58cbfff640 1 -- 172.21.15.110:0/2101499020 learned_addr learned my addr 172.21.15.110:0/2101499020 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:15.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.904+0000 7f58d0ff9640 10 client.?.objecter ms_handle_connect 0x7f58d40b4d60 2024-03-25T19:50:15.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.904+0000 7f58d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.925+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560225108 0 0) 0x7f58d41307c0 con 0x7f58d40b4d60 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.925+0000 7f58d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.925+0000 7f58d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.925+0000 7f58d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.925+0000 7f58d0ff9640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.925+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b4004eb0 con 0x7f58d40b4d60 2024-03-25T19:50:15.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.931+0000 7f58d0ff9640 10 client.?.objecter ms_handle_connect 0x7f58d40b1730 2024-03-25T19:50:15.932 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.931+0000 7f58d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:15.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.938+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059494900 0 0) 0x7f58d4138f90 con 0x7f58d40b1730 2024-03-25T19:50:15.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.938+0000 7f58d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:15.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.938+0000 7f58d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:15.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.938+0000 7f58d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:15.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.938+0000 7f58d0ff9640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:50:15.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.938+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d41307c0 con 0x7f58d40b1730 2024-03-25T19:50:15.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.938+0000 7f58d0ff9640 10 client.?.objecter ms_handle_connect 0x7f58d414be90 2024-03-25T19:50:15.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.938+0000 7f58d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:15.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.950+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103982032 0 0) 0x7f58d4132ab0 con 0x7f58d414be90 2024-03-25T19:50:15.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.950+0000 7f58d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:15.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.950+0000 7f58d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:15.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.950+0000 7f58d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:15.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.950+0000 7f58d0ff9640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:50:15.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.950+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d4138f90 con 0x7f58d414be90 2024-03-25T19:50:15.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.950+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3509476874 0 0) 0x7f58d41307c0 con 0x7f58d40b1730 2024-03-25T19:50:15.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.951+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58d4132ab0 con 0x7f58d40b1730 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.951+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58c40018a0 con 0x7f58d40b1730 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.951+0000 7f58d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.951+0000 7f58d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.951+0000 7f58d0ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:15.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.951+0000 7f58d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.953+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1087481402 0 0) 0x7f58b4004eb0 con 0x7f58d40b4d60 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.953+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58d41307c0 con 0x7f58d40b4d60 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.953+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58bc0019d0 con 0x7f58d40b4d60 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.953+0000 7f58d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.953+0000 7f58d0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.953+0000 7f58d0ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.954 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:15.955 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:15.955 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:15.955 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:15.955 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3214286031 0 0) 0x7f58d41307c0 con 0x7f58d40b4d60 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 >> v1:172.21.15.110:6790/0 conn(0x7f58d414be90 legacy=0x7f58d41300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 >> v1:172.21.15.178:6789/0 conn(0x7f58d40b1730 legacy=0x7f58d41393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 1 monclient: found mon.a 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4133a40 con 0x7f58d40b4d60 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:15.968 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 5 monclient: authenticate success, global_id 4668 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 10 monclient: _renew_subs 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d41316c0 con 0x7f58d40b4d60 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 10 client.4668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 10 monclient: _renew_subs 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d4137860 con 0x7f58d40b4d60 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.967+0000 7f58d9cfb640 1 librados: init done 2024-03-25T19:50:15.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.968+0000 7f58d9cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:15.970 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.968+0000 7f58d9cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:15.970 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.968+0000 7f58d9cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:15.970 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:15.968+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d4137bb0 con 0x7f58d40b4d60 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.295+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58bc0035f0 con 0x7f58d40b4d60 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.295+0000 7f58d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.295+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f58bc003820 con 0x7f58d40b4d60 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.295+0000 7f58d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.295+0000 7f58d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.295+0000 7f58d0ff9640 10 monclient: dump: 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:16.296 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:16.297 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:16.297 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:16.297 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:16.297 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:16.297 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.295+0000 7f58d27fc640 4 set_mon_vals no callback set 2024-03-25T19:50:16.297 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.295+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f58bc003a40 con 0x7f58d40b4d60 2024-03-25T19:50:16.297 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.296+0000 7f58d37fe640 10 monclient: get_auth_request con 0x7f58b406dfb0 auth_method 0 2024-03-25T19:50:16.298 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.296+0000 7f58d0ff9640 10 client.4668.objecter ms_handle_connect 0x7f58b406dfb0 2024-03-25T19:50:16.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.301+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7f58bc0545d0 con 0x7f58d40b4d60 2024-03-25T19:50:16.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.301+0000 7f58d0ff9640 10 client.4668.objecter ms_dispatch 0x7f58d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:16.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.301+0000 7f58d0ff9640 3 client.4668.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:16.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.302+0000 7f58d0ff9640 3 client.4668.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:16.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.302+0000 7f58d0ff9640 20 client.4668.objecter dump_active .. 0 homeless 2024-03-25T19:50:16.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.302+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58bc0500e0 con 0x7f58d40b4d60 2024-03-25T19:50:16.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.302+0000 7f58d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:16.303 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.302+0000 7f58d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:16.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.562+0000 7f58d9cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:16.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.562+0000 7f58d9cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:16.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.562+0000 7f58d9cfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:16.563 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.562+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58d41319b0 con 0x7f58d40b4d60 2024-03-25T19:50:16.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.625+0000 7f58d0ff9640 1 -- 172.21.15.110:0/2101499020 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3733485007) 0x7f58bc052010 con 0x7f58d40b4d60 2024-03-25T19:50:16.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.625+0000 7f58d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:16.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.625+0000 7f58d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:16.626 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:50:16.626 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:40:25.624450+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:25.624450+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:16.628 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 10 librados: watch_flush enter 2024-03-25T19:50:16.628 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 10 librados: watch_flush exit 2024-03-25T19:50:16.628 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 20 client.4668.objecter shutdown clearing up homeless session... 2024-03-25T19:50:16.628 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 10 client.4668.objecter successfully canceled tick 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 >> v1:172.21.15.178:6800/1132739058 conn(0x7f58b406dfb0 legacy=0x7f58b4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 10 monclient: shutdown 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 >> v1:172.21.15.110:6789/0 conn(0x7f58d40b4d60 legacy=0x7f58d412f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 shutdown_connections 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 >> 172.21.15.110:0/2101499020 conn(0x7f58d40617a0 msgr2=0x7f58d413c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 shutdown_connections 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 1 -- 172.21.15.110:0/2101499020 wait complete. 2024-03-25T19:50:16.629 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:16.627+0000 7f58d9cfb640 1 librados: shutdown 2024-03-25T19:50:16.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:23.448 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:23.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.562+0000 7efd4d86e640 10 monclient: get_monmap_and_config 2024-03-25T19:50:23.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.562+0000 7efd4d86e640 10 monclient: build_initial_monmap 2024-03-25T19:50:23.562 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.562+0000 7efd4d86e640 10 monclient: monmap: 2024-03-25T19:50:23.562 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:23.562 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:23.562 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:50:23.563388+0000 2024-03-25T19:50:23.562 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:50:23.563388+0000 2024-03-25T19:50:23.562 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:23.563 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:23.563 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:23.563 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:50:23.563 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:50:23.563 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:23.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.563+0000 7efd4d86e640 1 Processor -- start 2024-03-25T19:50:23.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.563+0000 7efd4d86e640 1 -- start start 2024-03-25T19:50:23.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.563+0000 7efd4d86e640 10 monclient: init 2024-03-25T19:50:23.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:23.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:23.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 10 monclient(hunting): picked mon.noname-b con 0x7efd4806ea40 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:23.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 10 monclient(hunting): picked mon.noname-c con 0x7efd48071800 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:23.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 10 monclient(hunting): picked mon.noname-a con 0x7efd48071c40 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:23.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4806f1f0 con 0x7efd48071c40 2024-03-25T19:50:23.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd48070d00 con 0x7efd48071800 2024-03-25T19:50:23.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd480720b0 con 0x7efd4806ea40 2024-03-25T19:50:23.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:23.565 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.564+0000 7efd4d86e640 10 monclient(hunting): authenticate will time out at 4794.705048s 2024-03-25T19:50:23.569 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.570+0000 7efd46ffd640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7efd4806ea40 0x7efd4813ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:38244/0 (socket says 172.21.15.110:38244) 2024-03-25T19:50:23.570 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.570+0000 7efd46ffd640 1 -- 172.21.15.110:0/828841011 learned_addr learned my addr 172.21.15.110:0/828841011 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:23.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.576+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147802770 0 0) 0x7efd480720b0 con 0x7efd4806ea40 2024-03-25T19:50:23.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.576+0000 7efd467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:23.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.576+0000 7efd467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:23.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.576+0000 7efd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:23.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.576+0000 7efd467fc640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:50:23.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.576+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd34003520 con 0x7efd4806ea40 2024-03-25T19:50:23.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.590+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786744456 0 0) 0x7efd48070d00 con 0x7efd48071800 2024-03-25T19:50:23.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.590+0000 7efd467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:23.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.590+0000 7efd467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:23.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.590+0000 7efd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:23.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.590+0000 7efd467fc640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:50:23.591 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.590+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd480720b0 con 0x7efd48071800 2024-03-25T19:50:23.599 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3715901022 0 0) 0x7efd480720b0 con 0x7efd48071800 2024-03-25T19:50:23.599 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 >> v1:172.21.15.110:6790/0 conn(0x7efd4806ea40 legacy=0x7efd4813ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 >> v1:172.21.15.110:6789/0 conn(0x7efd48071c40 legacy=0x7efd4814a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 1 monclient: found mon.noname-c 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd48072780 con 0x7efd48071800 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efd30001670 con 0x7efd48071800 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd4d86e640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd4d86e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:23.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd467fc640 10 monclient: dump: 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:23.601 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.599+0000 7efd4d86e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:23.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.620+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd300017d0 con 0x7efd48071800 2024-03-25T19:50:23.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.620+0000 7efd467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:23.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.620+0000 7efd467fc640 1 -- 172.21.15.110:0/828841011 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efd30001b60 con 0x7efd48071800 2024-03-25T19:50:23.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.620+0000 7efd467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:23.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.620+0000 7efd467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:23.620 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.620+0000 7efd467fc640 10 monclient: dump: 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.620+0000 7efd4d86e640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:23.621 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.620+0000 7efd4d86e640 4 set_mon_vals no callback set 2024-03-25T19:50:23.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.621+0000 7efd4d86e640 10 monclient: shutdown 2024-03-25T19:50:23.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.621+0000 7efd4d86e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:23.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.621+0000 7efd4d86e640 1 -- 172.21.15.110:0/828841011 >> v1:172.21.15.178:6789/0 conn(0x7efd48071800 legacy=0x7efd48147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.622+0000 7efd4d86e640 1 -- 172.21.15.110:0/828841011 shutdown_connections 2024-03-25T19:50:23.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.622+0000 7efd4d86e640 1 -- 172.21.15.110:0/828841011 >> 172.21.15.110:0/828841011 conn(0x7efd4806b630 msgr2=0x7efd480b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:23.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.622+0000 7efd4d86e640 1 -- 172.21.15.110:0/828841011 shutdown_connections 2024-03-25T19:50:23.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.622+0000 7efd4d86e640 1 -- 172.21.15.110:0/828841011 wait complete. 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.622+0000 7efd4d86e640 10 monclient: build_initial_monmap 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.622+0000 7efd4d86e640 10 monclient: monmap: 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:50:23.623 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.623+0000 7efd4d86e640 1 librados: starting msgr at 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.623+0000 7efd4d86e640 1 librados: starting objecter 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.623+0000 7efd4d86e640 1 Processor -- start 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.623+0000 7efd4d86e640 1 -- start start 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.623+0000 7efd4d86e640 1 librados: setting wanted keys 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.623+0000 7efd4d86e640 1 librados: calling monclient init 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.623+0000 7efd4d86e640 10 monclient: init 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:23.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:23.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 10 monclient(hunting): picked mon.noname-b con 0x7efd4806ea40 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:23.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 10 monclient(hunting): picked mon.noname-a con 0x7efd48071800 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:23.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 10 monclient(hunting): picked mon.noname-c con 0x7efd48071c40 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:23.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd481c57d0 con 0x7efd48071800 2024-03-25T19:50:23.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd481d7140 con 0x7efd4806ea40 2024-03-25T19:50:23.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd481d6d10 con 0x7efd48071c40 2024-03-25T19:50:23.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:23.626 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.624+0000 7efd4d86e640 10 monclient(hunting): authenticate will time out at 4794.765688s 2024-03-25T19:50:23.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.638+0000 7efd477fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7efd48071c40 0x7efd480acaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:38290/0 (socket says 172.21.15.110:38290) 2024-03-25T19:50:23.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.638+0000 7efd477fe640 1 -- 172.21.15.110:0/1356093300 learned_addr learned my addr 172.21.15.110:0/1356093300 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:23.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.639+0000 7efd44ff9640 10 client.?.objecter ms_handle_connect 0x7efd48071c40 2024-03-25T19:50:23.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.639+0000 7efd44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:23.646 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.647+0000 7efd44ff9640 10 client.?.objecter ms_handle_connect 0x7efd4806ea40 2024-03-25T19:50:23.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.647+0000 7efd44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:23.648 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.649+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014477669 0 0) 0x7efd481d6d10 con 0x7efd48071c40 2024-03-25T19:50:23.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.649+0000 7efd44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:23.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.649+0000 7efd44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:23.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.649+0000 7efd44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:23.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.649+0000 7efd44ff9640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:50:23.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.649+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd28004eb0 con 0x7efd48071c40 2024-03-25T19:50:23.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.650+0000 7efd44ff9640 10 client.?.objecter ms_handle_connect 0x7efd48071800 2024-03-25T19:50:23.649 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.650+0000 7efd44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.658+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 418642201 0 0) 0x7efd28004eb0 con 0x7efd48071c40 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.659+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd481d6d10 con 0x7efd48071c40 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.659+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efd40001c90 con 0x7efd48071c40 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.659+0000 7efd44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.659+0000 7efd44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.659+0000 7efd44ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:23.659 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:23.660 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:23.660 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:23.660 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:23.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.659+0000 7efd44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:23.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.662+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964650921 0 0) 0x7efd481c57d0 con 0x7efd48071800 2024-03-25T19:50:23.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.662+0000 7efd44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:23.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.662+0000 7efd44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:23.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.662+0000 7efd44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:23.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.662+0000 7efd44ff9640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:50:23.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.662+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd28004eb0 con 0x7efd48071800 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.738+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3192892245 0 0) 0x7efd28004eb0 con 0x7efd48071800 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.738+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd481c57d0 con 0x7efd48071800 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.738+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efd30001ae0 con 0x7efd48071800 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.738+0000 7efd44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.738+0000 7efd44ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.738+0000 7efd44ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:23.738 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:23.739 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:23.739 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:23.739 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:23.739 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:23.739 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:23.739 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:23.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 651443515 0 0) 0x7efd481c57d0 con 0x7efd48071800 2024-03-25T19:50:23.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 >> v1:172.21.15.110:6790/0 conn(0x7efd48071c40 legacy=0x7efd480acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 >> v1:172.21.15.178:6789/0 conn(0x7efd4806ea40 legacy=0x7efd481d6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:23.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 1 monclient: found mon.a 2024-03-25T19:50:23.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:23.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:23.760 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd481d6ea0 con 0x7efd48071800 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 5 monclient: authenticate success, global_id 4674 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 10 monclient: _renew_subs 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd481c8db0 con 0x7efd48071800 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 10 client.4674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 10 monclient: _renew_subs 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd481de970 con 0x7efd48071800 2024-03-25T19:50:23.761 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.760+0000 7efd4d86e640 1 librados: init done 2024-03-25T19:50:23.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.761+0000 7efd4d86e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:23.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.761+0000 7efd4d86e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:23.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.761+0000 7efd4d86e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:23.762 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:23.761+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd48074d80 con 0x7efd48071800 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.104+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd300038a0 con 0x7efd48071800 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.104+0000 7efd44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.104+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7efd30003a10 con 0x7efd48071800 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.104+0000 7efd44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.104+0000 7efd44ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.104+0000 7efd44ff9640 10 monclient: dump: 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:24.104 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:24.105 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:24.105 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:24.105 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:24.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.104+0000 7efd45ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:24.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.105+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7efd300199e0 con 0x7efd48071800 2024-03-25T19:50:24.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.105+0000 7efd46ffd640 10 monclient: get_auth_request con 0x7efd28070040 auth_method 0 2024-03-25T19:50:24.105 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.106+0000 7efd44ff9640 10 client.4674.objecter ms_handle_connect 0x7efd28070040 2024-03-25T19:50:24.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.112+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2953492977 0 0) 0x7efd300534a0 con 0x7efd48071800 2024-03-25T19:50:24.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.112+0000 7efd44ff9640 10 client.4674.objecter ms_dispatch 0x7efd48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:24.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.112+0000 7efd44ff9640 3 client.4674.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:24.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.112+0000 7efd44ff9640 3 client.4674.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:24.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.112+0000 7efd44ff9640 20 client.4674.objecter dump_active .. 0 homeless 2024-03-25T19:50:24.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.112+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd3004efb0 con 0x7efd48071800 2024-03-25T19:50:24.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.112+0000 7efd44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:24.112 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.112+0000 7efd44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:24.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.342+0000 7efd4d86e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:24.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.342+0000 7efd4d86e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:24.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.342+0000 7efd4d86e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:24.342 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.342+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd480627b0 con 0x7efd48071800 2024-03-25T19:50:24.342 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:31.626 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:24.387 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:31.641 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:24.391 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:31.645 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:24.397 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:31.681 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:24.424 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:31.679 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:24.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.431+0000 7efd44ff9640 1 -- 172.21.15.110:0/1356093300 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7efd30050ee0 con 0x7efd48071800 2024-03-25T19:50:24.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.431+0000 7efd44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:24.431 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.431+0000 7efd44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:24.431 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:50:24.431 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:24.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.433+0000 7efd4d86e640 10 librados: watch_flush enter 2024-03-25T19:50:24.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 10 librados: watch_flush exit 2024-03-25T19:50:24.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 20 client.4674.objecter shutdown clearing up homeless session... 2024-03-25T19:50:24.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 10 client.4674.objecter successfully canceled tick 2024-03-25T19:50:24.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 >> v1:172.21.15.178:6800/1132739058 conn(0x7efd28070040 legacy=0x7efd28072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:24.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 10 monclient: shutdown 2024-03-25T19:50:24.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:24.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 >> v1:172.21.15.110:6789/0 conn(0x7efd48071800 legacy=0x7efd480ac3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:24.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 shutdown_connections 2024-03-25T19:50:24.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 >> 172.21.15.110:0/1356093300 conn(0x7efd4806b630 msgr2=0x7efd480af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:24.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 shutdown_connections 2024-03-25T19:50:24.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 1 -- 172.21.15.110:0/1356093300 wait complete. 2024-03-25T19:50:24.435 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:24.434+0000 7efd4d86e640 1 librados: shutdown 2024-03-25T19:50:24.439 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:31.723 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:24.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:24.445 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:31.730 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:24.484 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:31.768 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:24.520 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:31.804 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:31.250 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:31.365 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.365+0000 7faef8554640 10 monclient: get_monmap_and_config 2024-03-25T19:50:31.365 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.365+0000 7faef8554640 10 monclient: build_initial_monmap 2024-03-25T19:50:31.365 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.365+0000 7faef8554640 10 monclient: monmap: 2024-03-25T19:50:31.365 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:31.365 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:31.365 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:50:31.366384+0000 2024-03-25T19:50:31.365 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:50:31.366384+0000 2024-03-25T19:50:31.365 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:31.366 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:31.366 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:31.366 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:50:31.366 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:50:31.366 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:31.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.366+0000 7faef8554640 1 Processor -- start 2024-03-25T19:50:31.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.366+0000 7faef8554640 1 -- start start 2024-03-25T19:50:31.366 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.366+0000 7faef8554640 10 monclient: init 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 10 monclient(hunting): picked mon.noname-c con 0x7faef00b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 10 monclient(hunting): picked mon.noname-b con 0x7faef00b1730 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 10 monclient(hunting): picked mon.noname-a con 0x7faef014be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef00af940 con 0x7faef014be90 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef00b0a70 con 0x7faef00b4d60 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef00b0d50 con 0x7faef00b1730 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:31.367 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.367+0000 7faef8554640 10 monclient(hunting): authenticate will time out at 4802.508422s 2024-03-25T19:50:31.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.371+0000 7faef5aef640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7faef00b1730 0x7faef00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:42900/0 (socket says 172.21.15.110:42900) 2024-03-25T19:50:31.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.371+0000 7faef5aef640 1 -- 172.21.15.110:0/222612924 learned_addr learned my addr 172.21.15.110:0/222612924 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:31.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.390+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993825215 0 0) 0x7faef00af940 con 0x7faef014be90 2024-03-25T19:50:31.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.390+0000 7faef52ee640 10 monclient(hunting): _negotiate 2024-03-25T19:50:31.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.390+0000 7faef52ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:31.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.390+0000 7faef52ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:31.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.390+0000 7faef52ee640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:50:31.390 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.390+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faedc003520 con 0x7faef014be90 2024-03-25T19:50:31.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.391+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379361865 0 0) 0x7faef00b0d50 con 0x7faef00b1730 2024-03-25T19:50:31.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.391+0000 7faef52ee640 10 monclient(hunting): _negotiate 2024-03-25T19:50:31.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.391+0000 7faef52ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:31.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.391+0000 7faef52ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:31.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.391+0000 7faef52ee640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:50:31.393 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.391+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef00af940 con 0x7faef00b1730 2024-03-25T19:50:31.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.394+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303304122 0 0) 0x7faef00b0a70 con 0x7faef00b4d60 2024-03-25T19:50:31.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.394+0000 7faef52ee640 10 monclient(hunting): _negotiate 2024-03-25T19:50:31.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.394+0000 7faef52ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:31.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.394+0000 7faef52ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:31.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.394+0000 7faef52ee640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:50:31.395 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.394+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef00b0d50 con 0x7faef00b4d60 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704517765 0 0) 0x7faef00af940 con 0x7faef00b1730 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 >> v1:172.21.15.178:6789/0 conn(0x7faef00b4d60 legacy=0x7faef013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 >> v1:172.21.15.110:6789/0 conn(0x7faef014be90 legacy=0x7faef014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 10 monclient: _finish_hunting 0 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 1 monclient: found mon.noname-b 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef00b1030 con 0x7faef00b1730 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 10 monclient: _finish_auth 0 2024-03-25T19:50:31.407 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 10 monclient: _check_auth_tickets 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef8554640 5 monclient: authenticate success, global_id 4685 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef8554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faee0001670 con 0x7faef00b1730 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef52ee640 10 monclient: dump: 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:31.408 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:31.409 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.407+0000 7faef8554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:31.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.420+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faee00017d0 con 0x7faef00b1730 2024-03-25T19:50:31.419 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.420+0000 7faef52ee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.420+0000 7faef52ee640 1 -- 172.21.15.110:0/222612924 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faee0001be0 con 0x7faef00b1730 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.420+0000 7faef52ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.420+0000 7faef52ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.420+0000 7faef52ee640 10 monclient: dump: 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.420+0000 7faef8554640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:31.420 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.420+0000 7faef8554640 4 set_mon_vals no callback set 2024-03-25T19:50:31.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.421+0000 7faef8554640 10 monclient: shutdown 2024-03-25T19:50:31.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.421+0000 7faef8554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:31.421 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.421+0000 7faef8554640 1 -- 172.21.15.110:0/222612924 >> v1:172.21.15.110:6790/0 conn(0x7faef00b1730 legacy=0x7faef00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.421+0000 7faef8554640 1 -- 172.21.15.110:0/222612924 shutdown_connections 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.421+0000 7faef8554640 1 -- 172.21.15.110:0/222612924 >> 172.21.15.110:0/222612924 conn(0x7faef00617a0 msgr2=0x7faef0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.421+0000 7faef8554640 1 -- 172.21.15.110:0/222612924 shutdown_connections 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.421+0000 7faef8554640 1 -- 172.21.15.110:0/222612924 wait complete. 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 10 monclient: build_initial_monmap 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 10 monclient: monmap: 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:50:31.422 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:31.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 1 librados: starting msgr at 2024-03-25T19:50:31.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 1 librados: starting objecter 2024-03-25T19:50:31.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 1 Processor -- start 2024-03-25T19:50:31.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 1 -- start start 2024-03-25T19:50:31.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 1 librados: setting wanted keys 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 1 librados: calling monclient init 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.422+0000 7faef8554640 10 monclient: init 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 10 monclient(hunting): picked mon.noname-a con 0x7faef00b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 10 monclient(hunting): picked mon.noname-b con 0x7faef00b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 10 monclient(hunting): picked mon.noname-c con 0x7faef014be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef00a9610 con 0x7faef00b1730 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef01433d0 con 0x7faef00b4d60 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef0143560 con 0x7faef014be90 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:31.424 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.423+0000 7faef8554640 10 monclient(hunting): authenticate will time out at 4802.564445s 2024-03-25T19:50:31.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.434+0000 7faef5aef640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7faef00b4d60 0x7faef01d80b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:49050/0 (socket says 172.21.15.110:49050) 2024-03-25T19:50:31.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.434+0000 7faef5aef640 1 -- 172.21.15.110:0/828438711 learned_addr learned my addr 172.21.15.110:0/828438711 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:31.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.436+0000 7faee6ffd640 10 client.?.objecter ms_handle_connect 0x7faef00b4d60 2024-03-25T19:50:31.436 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.436+0000 7faee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:31.444 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.445+0000 7faee6ffd640 10 client.?.objecter ms_handle_connect 0x7faef00b1730 2024-03-25T19:50:31.445 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.445+0000 7faee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:31.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.446+0000 7faee6ffd640 10 client.?.objecter ms_handle_connect 0x7faef014be90 2024-03-25T19:50:31.446 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.446+0000 7faee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:31.450 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.450+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072446970 0 0) 0x7faef01433d0 con 0x7faef00b4d60 2024-03-25T19:50:31.450 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.450+0000 7faee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:31.450 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.450+0000 7faee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:31.450 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.450+0000 7faee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:31.450 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.450+0000 7faee6ffd640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:50:31.450 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.450+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed4004eb0 con 0x7faef00b4d60 2024-03-25T19:50:31.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.461+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271745316 0 0) 0x7faef00a9610 con 0x7faef00b1730 2024-03-25T19:50:31.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.461+0000 7faee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:31.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.461+0000 7faee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:31.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.461+0000 7faee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:31.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.461+0000 7faee6ffd640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:50:31.461 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.461+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef01433d0 con 0x7faef00b1730 2024-03-25T19:50:31.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.469+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3362573994 0 0) 0x7faed4004eb0 con 0x7faef00b4d60 2024-03-25T19:50:31.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.469+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faef00a9610 con 0x7faef00b4d60 2024-03-25T19:50:31.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.469+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faee0001c20 con 0x7faef00b4d60 2024-03-25T19:50:31.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.469+0000 7faee6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:31.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.469+0000 7faee6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:31.469 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.469+0000 7faee6ffd640 10 monclient(hunting): dump: 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:31.470 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.469+0000 7faee6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:31.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.482+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525801102 0 0) 0x7faef0143560 con 0x7faef014be90 2024-03-25T19:50:31.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.482+0000 7faee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:31.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.482+0000 7faee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:31.482 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.482+0000 7faee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:31.483 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.482+0000 7faee6ffd640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:50:31.483 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.482+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed4004eb0 con 0x7faef014be90 2024-03-25T19:50:31.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2721137318 0 0) 0x7faef00a9610 con 0x7faef00b4d60 2024-03-25T19:50:31.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 >> v1:172.21.15.110:6790/0 conn(0x7faef014be90 legacy=0x7faef01e2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:31.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 >> v1:172.21.15.110:6789/0 conn(0x7faef00b1730 legacy=0x7faef01d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:31.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:31.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 1 monclient: found mon.b 2024-03-25T19:50:31.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:31.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:31.485 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef01436f0 con 0x7faef00b4d60 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 5 monclient: authenticate success, global_id 4687 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 10 monclient: _renew_subs 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef01429a0 con 0x7faef00b4d60 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 10 monclient: _renew_subs 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faef0140560 con 0x7faef00b4d60 2024-03-25T19:50:31.486 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.485+0000 7faef8554640 1 librados: init done 2024-03-25T19:50:31.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.486+0000 7faef8554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:31.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.486+0000 7faef8554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:31.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.486+0000 7faef8554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:31.487 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.486+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef0061d10 con 0x7faef00b4d60 2024-03-25T19:50:31.771 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.772+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faee0004a90 con 0x7faef00b4d60 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.772+0000 7faee6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.772+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7faee0004de0 con 0x7faef00b4d60 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.772+0000 7faee6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.772+0000 7faee6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.772+0000 7faee6ffd640 10 monclient: dump: 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:31.772 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:31.773 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:31.773 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:31.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.772+0000 7faef4aed640 4 set_mon_vals no callback set 2024-03-25T19:50:31.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.772+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7faee0005000 con 0x7faef00b4d60 2024-03-25T19:50:31.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.773+0000 7faef62f0640 10 monclient: get_auth_request con 0x7faed406dd60 auth_method 0 2024-03-25T19:50:31.773 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.773+0000 7faee6ffd640 10 client.4687.objecter ms_handle_connect 0x7faed406dd60 2024-03-25T19:50:31.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.777+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7faee0054950 con 0x7faef00b4d60 2024-03-25T19:50:31.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.778+0000 7faee6ffd640 10 client.4687.objecter ms_dispatch 0x7faef0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:50:31.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.778+0000 7faee6ffd640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:50:31.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.778+0000 7faee6ffd640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:50:31.777 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.778+0000 7faee6ffd640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T19:50:31.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.779+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faee0050460 con 0x7faef00b4d60 2024-03-25T19:50:31.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.779+0000 7faee6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:31.778 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:31.779+0000 7faee6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:32.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.003+0000 7faef8554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:32.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.003+0000 7faef8554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:32.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.003+0000 7faef8554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:32.006 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.003+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faef00a97a0 con 0x7faef00b4d60 2024-03-25T19:50:32.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.034+0000 7faee6ffd640 1 -- 172.21.15.110:0/828438711 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7faee0052390 con 0x7faef00b4d60 2024-03-25T19:50:32.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.034+0000 7faee6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:32.034 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.034+0000 7faee6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:32.034 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:50:32.034 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:32.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 10 librados: watch_flush enter 2024-03-25T19:50:32.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 10 librados: watch_flush exit 2024-03-25T19:50:32.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T19:50:32.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 10 client.4687.objecter successfully canceled tick 2024-03-25T19:50:32.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 >> v1:172.21.15.178:6800/1132739058 conn(0x7faed406dd60 legacy=0x7faed4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:32.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 10 monclient: shutdown 2024-03-25T19:50:32.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:32.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 >> v1:172.21.15.178:6789/0 conn(0x7faef00b4d60 legacy=0x7faef01d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:32.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 shutdown_connections 2024-03-25T19:50:32.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 >> 172.21.15.110:0/828438711 conn(0x7faef00617a0 msgr2=0x7faef013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:32.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 shutdown_connections 2024-03-25T19:50:32.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 1 -- 172.21.15.110:0/828438711 wait complete. 2024-03-25T19:50:32.037 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:32.036+0000 7faef8554640 1 librados: shutdown 2024-03-25T19:50:32.048 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:38.855 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.964+0000 7f6f107c7640 10 monclient: get_monmap_and_config 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.964+0000 7f6f107c7640 10 monclient: build_initial_monmap 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.964+0000 7f6f107c7640 10 monclient: monmap: 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:50:38.965505+0000 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:50:38.965505+0000 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:38.964 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:38.965 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:50:38.965 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:50:38.965 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:38.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.965+0000 7f6f107c7640 1 Processor -- start 2024-03-25T19:50:38.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.965+0000 7f6f107c7640 1 -- start start 2024-03-25T19:50:38.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.965+0000 7f6f107c7640 10 monclient: init 2024-03-25T19:50:38.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:38.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:38.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 10 monclient(hunting): picked mon.noname-a con 0x7f6f080a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:38.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:38.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 10 monclient(hunting): picked mon.noname-b con 0x7f6f081430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:38.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f08142520 con 0x7f6f080a98f0 2024-03-25T19:50:38.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080b28a0 con 0x7f6f0813f710 2024-03-25T19:50:38.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080b2a30 con 0x7f6f081430c0 2024-03-25T19:50:38.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:38.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.966+0000 7f6f107c7640 10 monclient(hunting): authenticate will time out at 4810.107208s 2024-03-25T19:50:38.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.969+0000 7f6f0dd62640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f6f0813f710 0x7f6f081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:40228/0 (socket says 172.21.15.110:40228) 2024-03-25T19:50:38.969 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.969+0000 7f6f0dd62640 1 -- 172.21.15.110:0/3500456129 learned_addr learned my addr 172.21.15.110:0/3500456129 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:38.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.979+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036324285 0 0) 0x7f6f080b28a0 con 0x7f6f0813f710 2024-03-25T19:50:38.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.979+0000 7f6f0d561640 10 monclient(hunting): _negotiate 2024-03-25T19:50:38.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.979+0000 7f6f0d561640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:38.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.979+0000 7f6f0d561640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:38.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.979+0000 7f6f0d561640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:50:38.979 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.979+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef4003520 con 0x7f6f0813f710 2024-03-25T19:50:38.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.982+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340359140 0 0) 0x7f6f08142520 con 0x7f6f080a98f0 2024-03-25T19:50:38.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.982+0000 7f6f0d561640 10 monclient(hunting): _negotiate 2024-03-25T19:50:38.981 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.982+0000 7f6f0d561640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:38.982 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.982+0000 7f6f0d561640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:38.982 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.982+0000 7f6f0d561640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:50:38.982 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.982+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f080b28a0 con 0x7f6f080a98f0 2024-03-25T19:50:38.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762096443 0 0) 0x7f6f080b28a0 con 0x7f6f080a98f0 2024-03-25T19:50:38.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 >> v1:172.21.15.110:6790/0 conn(0x7f6f081430c0 legacy=0x7f6f0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:38.997 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 >> v1:172.21.15.178:6789/0 conn(0x7f6f0813f710 legacy=0x7f6f081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 10 monclient: _finish_hunting 0 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 1 monclient: found mon.noname-a 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f080b44c0 con 0x7f6f080a98f0 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 10 monclient: _finish_auth 0 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 10 monclient: _check_auth_tickets 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:38.998 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6f04001670 con 0x7f6f080a98f0 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f0d561640 10 monclient: dump: 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f107c7640 5 monclient: authenticate success, global_id 4683 2024-03-25T19:50:38.999 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:38.997+0000 7f6f107c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:39.023 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.023+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f040017d0 con 0x7f6f080a98f0 2024-03-25T19:50:39.023 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.023+0000 7f6f0d561640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:39.023 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.023+0000 7f6f0d561640 1 -- 172.21.15.110:0/3500456129 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6f04001b60 con 0x7f6f080a98f0 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.023+0000 7f6f107c7640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.024+0000 7f6f0d561640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.024+0000 7f6f0d561640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.024+0000 7f6f0d561640 10 monclient: dump: 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:39.024 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.024+0000 7f6f107c7640 4 set_mon_vals no callback set 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.024+0000 7f6f107c7640 10 monclient: shutdown 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.024+0000 7f6f107c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.024+0000 7f6f107c7640 1 -- 172.21.15.110:0/3500456129 >> v1:172.21.15.110:6789/0 conn(0x7f6f080a98f0 legacy=0x7f6f080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.025+0000 7f6f107c7640 1 -- 172.21.15.110:0/3500456129 shutdown_connections 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.025+0000 7f6f107c7640 1 -- 172.21.15.110:0/3500456129 >> 172.21.15.110:0/3500456129 conn(0x7f6f080b7ea0 msgr2=0x7f6f080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.025+0000 7f6f107c7640 1 -- 172.21.15.110:0/3500456129 shutdown_connections 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.025+0000 7f6f107c7640 1 -- 172.21.15.110:0/3500456129 wait complete. 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 10 monclient: build_initial_monmap 2024-03-25T19:50:39.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 10 monclient: monmap: 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 1 librados: starting msgr at 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 1 librados: starting objecter 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 1 Processor -- start 2024-03-25T19:50:39.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 1 -- start start 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 1 librados: setting wanted keys 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 1 librados: calling monclient init 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.026+0000 7f6f107c7640 10 monclient: init 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 10 monclient(hunting): picked mon.noname-c con 0x7f6f080a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 10 monclient(hunting): picked mon.noname-a con 0x7f6f0813f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 10 monclient(hunting): picked mon.noname-b con 0x7f6f081430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:39.029 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0813b3d0 con 0x7f6f0813f710 2024-03-25T19:50:39.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080b6e00 con 0x7f6f081430c0 2024-03-25T19:50:39.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f08139e50 con 0x7f6f080a98f0 2024-03-25T19:50:39.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:39.030 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.027+0000 7f6f107c7640 10 monclient(hunting): authenticate will time out at 4810.168411s 2024-03-25T19:50:39.035 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.036+0000 7f6f0ed64640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f6f081430c0 0x7f6f081e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:40280/0 (socket says 172.21.15.110:40280) 2024-03-25T19:50:39.036 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.036+0000 7f6f0ed64640 1 -- 172.21.15.110:0/1870383640 learned_addr learned my addr 172.21.15.110:0/1870383640 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:39.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.039+0000 7f6efeffd640 10 client.?.objecter ms_handle_connect 0x7f6f081430c0 2024-03-25T19:50:39.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.039+0000 7f6efeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:39.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.055+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680435918 0 0) 0x7f6f080b6e00 con 0x7f6f081430c0 2024-03-25T19:50:39.054 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.055+0000 7f6efeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:39.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.055+0000 7f6efeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:39.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.055+0000 7f6efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:39.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.055+0000 7f6efeffd640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:50:39.055 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.055+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec004eb0 con 0x7f6f081430c0 2024-03-25T19:50:39.059 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.060+0000 7f6efeffd640 10 client.?.objecter ms_handle_connect 0x7f6f0813f710 2024-03-25T19:50:39.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.060+0000 7f6efeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:39.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.060+0000 7f6efeffd640 10 client.?.objecter ms_handle_connect 0x7f6f080a98f0 2024-03-25T19:50:39.060 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.060+0000 7f6efeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:39.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.067+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930921620 0 0) 0x7f6f0813b3d0 con 0x7f6f0813f710 2024-03-25T19:50:39.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.067+0000 7f6efeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:39.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.067+0000 7f6efeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:39.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.067+0000 7f6efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:39.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.067+0000 7f6efeffd640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:50:39.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.067+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f080b6e00 con 0x7f6f0813f710 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.077+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2659105423 0 0) 0x7f6eec004eb0 con 0x7f6f081430c0 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.077+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f0813b3d0 con 0x7f6f081430c0 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.077+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6f00001c90 con 0x7f6f081430c0 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.077+0000 7f6efeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.077+0000 7f6efeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.077+0000 7f6efeffd640 10 monclient(hunting): dump: 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:39.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.077+0000 7f6efeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:39.088 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.089+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655512414 0 0) 0x7f6f08139e50 con 0x7f6f080a98f0 2024-03-25T19:50:39.088 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.089+0000 7f6efeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:39.088 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.089+0000 7f6efeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:39.088 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.089+0000 7f6efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:39.089 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.089+0000 7f6efeffd640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:50:39.089 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.089+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec004eb0 con 0x7f6f080a98f0 2024-03-25T19:50:39.096 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3954696428 0 0) 0x7f6f0813b3d0 con 0x7f6f081430c0 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 >> v1:172.21.15.110:6790/0 conn(0x7f6f080a98f0 legacy=0x7f6f080b75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 >> v1:172.21.15.110:6789/0 conn(0x7f6f0813f710 legacy=0x7f6f0812f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 1 monclient: found mon.b 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0812dd80 con 0x7f6f081430c0 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6efeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 10 monclient: _renew_subs 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f081397a0 con 0x7f6f081430c0 2024-03-25T19:50:39.097 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:39.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 10 monclient: _renew_subs 2024-03-25T19:50:39.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:39.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0813b9d0 con 0x7f6f081430c0 2024-03-25T19:50:39.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.096+0000 7f6f107c7640 1 librados: init done 2024-03-25T19:50:39.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.098+0000 7f6f107c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:39.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.098+0000 7f6f107c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:39.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.098+0000 7f6f107c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:39.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.098+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f08139fe0 con 0x7f6f081430c0 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.117+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f00003990 con 0x7f6f081430c0 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.117+0000 7f6efeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.117+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6f00003d60 con 0x7f6f081430c0 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.117+0000 7f6efeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.117+0000 7f6efeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.117+0000 7f6efeffd640 10 monclient: dump: 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:39.117 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:39.118 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:39.118 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:39.118 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:39.118 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:39.118 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:39.118 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:39.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.117+0000 7f6f0cd60640 4 set_mon_vals no callback set 2024-03-25T19:50:39.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.118+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f6f00003f80 con 0x7f6f081430c0 2024-03-25T19:50:39.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.118+0000 7f6f0e563640 10 monclient: get_auth_request con 0x7f6eec06de00 auth_method 0 2024-03-25T19:50:39.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.119+0000 7f6efeffd640 10 client.4693.objecter ms_handle_connect 0x7f6eec06de00 2024-03-25T19:50:39.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.121+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f6f00054ac0 con 0x7f6f081430c0 2024-03-25T19:50:39.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.121+0000 7f6efeffd640 10 client.4693.objecter ms_dispatch 0x7f6f08000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:50:39.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.121+0000 7f6efeffd640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:50:39.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.121+0000 7f6efeffd640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:50:39.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.121+0000 7f6efeffd640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T19:50:39.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.380+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f000505d0 con 0x7f6f081430c0 2024-03-25T19:50:39.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.380+0000 7f6efeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:39.380 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.380+0000 7f6efeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:39.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.622+0000 7f6f107c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:39.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.622+0000 7f6f107c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:39.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.622+0000 7f6f107c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:50:39.622 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.622+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f080627b0 con 0x7f6f081430c0 2024-03-25T19:50:39.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.651+0000 7f6efeffd640 1 -- 172.21.15.110:0/1870383640 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f6f00052500 con 0x7f6f081430c0 2024-03-25T19:50:39.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.651+0000 7f6efeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:39.651 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.651+0000 7f6efeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:39.651 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:50:39.651 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 10 librados: watch_flush enter 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 10 librados: watch_flush exit 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 10 client.4693.objecter successfully canceled tick 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 >> v1:172.21.15.178:6800/1132739058 conn(0x7f6eec06de00 legacy=0x7f6eec0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 10 monclient: shutdown 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 >> v1:172.21.15.178:6789/0 conn(0x7f6f081430c0 legacy=0x7f6f081e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 shutdown_connections 2024-03-25T19:50:39.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.653+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 >> 172.21.15.110:0/1870383640 conn(0x7f6f080b7ea0 msgr2=0x7f6f08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:39.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.654+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 shutdown_connections 2024-03-25T19:50:39.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.654+0000 7f6f107c7640 1 -- 172.21.15.110:0/1870383640 wait complete. 2024-03-25T19:50:39.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:39.654+0000 7f6f107c7640 1 librados: shutdown 2024-03-25T19:50:39.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:44.221 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:51.505 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:44.230 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:51.514 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:44.258 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:51.542 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:44.284 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:51.538 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:44.288 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:51.542 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:44.289 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:51.573 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:44.296 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:51.580 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:44.322 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:10:51.577 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:44.338 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:51.623 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.472 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:46.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.583+0000 7f6226390640 10 monclient: get_monmap_and_config 2024-03-25T19:50:46.582 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.583+0000 7f6226390640 10 monclient: build_initial_monmap 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.583+0000 7f6226390640 10 monclient: monmap: 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:50:46.584089+0000 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:50:46.584089+0000 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:50:46.583 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:46.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.583+0000 7f6226390640 1 Processor -- start 2024-03-25T19:50:46.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.583+0000 7f6226390640 1 -- start start 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.583+0000 7f6226390640 10 monclient: init 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 10 monclient(hunting): picked mon.noname-c con 0x7f62200b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 10 monclient(hunting): picked mon.noname-a con 0x7f62200b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 10 monclient(hunting): picked mon.noname-b con 0x7f622014be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62200af940 con 0x7f62200b1730 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62200b0a70 con 0x7f62200b4d60 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62200b0d50 con 0x7f622014be90 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:46.585 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.584+0000 7f6226390640 10 monclient(hunting): authenticate will time out at 4817.725584s 2024-03-25T19:50:46.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.593+0000 7f622492d640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f622014be90 0x7f622014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:44724/0 (socket says 172.21.15.110:44724) 2024-03-25T19:50:46.593 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.593+0000 7f622492d640 1 -- 172.21.15.110:0/2934674077 learned_addr learned my addr 172.21.15.110:0/2934674077 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:46.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.609+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197406113 0 0) 0x7f62200b0d50 con 0x7f622014be90 2024-03-25T19:50:46.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.609+0000 7f621effd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.609+0000 7f621effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.609+0000 7f621effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.609+0000 7f621effd640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:50:46.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.609+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6208003520 con 0x7f622014be90 2024-03-25T19:50:46.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.615+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189957830 0 0) 0x7f62200af940 con 0x7f62200b1730 2024-03-25T19:50:46.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.615+0000 7f621effd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.615+0000 7f621effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.615+0000 7f621effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.615+0000 7f621effd640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:50:46.615 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.615+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62200b0d50 con 0x7f62200b1730 2024-03-25T19:50:46.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.619+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664273842 0 0) 0x7f62200b0a70 con 0x7f62200b4d60 2024-03-25T19:50:46.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.619+0000 7f621effd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.619+0000 7f621effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.619+0000 7f621effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.619+0000 7f621effd640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:50:46.619 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.619+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62200af940 con 0x7f62200b4d60 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1151203521 0 0) 0x7f62200b0d50 con 0x7f62200b1730 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 >> v1:172.21.15.110:6790/0 conn(0x7f622014be90 legacy=0x7f622014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 >> v1:172.21.15.178:6789/0 conn(0x7f62200b4d60 legacy=0x7f622013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 1 monclient: found mon.noname-a 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62200b1030 con 0x7f62200b1730 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 10 monclient: _finish_auth 0 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6214001670 con 0x7f62200b1730 2024-03-25T19:50:46.624 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f621effd640 10 monclient: dump: 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f6226390640 5 monclient: authenticate success, global_id 4689 2024-03-25T19:50:46.625 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.624+0000 7f6226390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.633+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62140017d0 con 0x7f62200b1730 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.633+0000 7f621effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.633+0000 7f621effd640 1 -- 172.21.15.110:0/2934674077 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6214001be0 con 0x7f62200b1730 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.633+0000 7f621effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.633+0000 7f621effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.633+0000 7f621effd640 10 monclient: dump: 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:46.633 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:46.634 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:46.634 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:46.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.633+0000 7f6226390640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:46.634 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.633+0000 7f6226390640 4 set_mon_vals no callback set 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.634+0000 7f6226390640 10 monclient: shutdown 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.634+0000 7f6226390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.634+0000 7f6226390640 1 -- 172.21.15.110:0/2934674077 >> v1:172.21.15.110:6789/0 conn(0x7f62200b1730 legacy=0x7f62200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.634+0000 7f6226390640 1 -- 172.21.15.110:0/2934674077 shutdown_connections 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.634+0000 7f6226390640 1 -- 172.21.15.110:0/2934674077 >> 172.21.15.110:0/2934674077 conn(0x7f62200617a0 msgr2=0x7f6220061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.634+0000 7f6226390640 1 -- 172.21.15.110:0/2934674077 shutdown_connections 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.634+0000 7f6226390640 1 -- 172.21.15.110:0/2934674077 wait complete. 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.635+0000 7f6226390640 10 monclient: build_initial_monmap 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.635+0000 7f6226390640 10 monclient: monmap: 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:50:46.636 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:46.636 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:50:46.636 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:46.636 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:50:46.636 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:50:46.636 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:46.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.635+0000 7f6226390640 1 librados: starting msgr at 2024-03-25T19:50:46.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.635+0000 7f6226390640 1 librados: starting objecter 2024-03-25T19:50:46.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.635+0000 7f6226390640 1 Processor -- start 2024-03-25T19:50:46.637 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.636+0000 7f6226390640 1 -- start start 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.636+0000 7f6226390640 1 librados: setting wanted keys 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.636+0000 7f6226390640 1 librados: calling monclient init 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.636+0000 7f6226390640 10 monclient: init 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 10 monclient(hunting): picked mon.noname-b con 0x7f62200b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 10 monclient(hunting): picked mon.noname-c con 0x7f62200b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 10 monclient(hunting): picked mon.noname-a con 0x7f622014be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6220143d40 con 0x7f622014be90 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6220138480 con 0x7f62200b1730 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6220141130 con 0x7f62200b4d60 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.637+0000 7f6226390640 10 monclient(hunting): authenticate will time out at 4817.778662s 2024-03-25T19:50:46.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.655+0000 7f621f7fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f62200b4d60 0x7f6220142890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:44754/0 (socket says 172.21.15.110:44754) 2024-03-25T19:50:46.654 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.655+0000 7f621f7fe640 1 -- 172.21.15.110:0/428980261 learned_addr learned my addr 172.21.15.110:0/428980261 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:46.655 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.656+0000 7f621cff9640 10 client.?.objecter ms_handle_connect 0x7f62200b4d60 2024-03-25T19:50:46.656 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.656+0000 7f621cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:46.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.663+0000 7f621cff9640 10 client.?.objecter ms_handle_connect 0x7f622014be90 2024-03-25T19:50:46.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.663+0000 7f621cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:46.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.663+0000 7f621cff9640 10 client.?.objecter ms_handle_connect 0x7f62200b1730 2024-03-25T19:50:46.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.663+0000 7f621cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:46.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.669+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695116903 0 0) 0x7f6220141130 con 0x7f62200b4d60 2024-03-25T19:50:46.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.669+0000 7f621cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.669+0000 7f621cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.669+0000 7f621cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.669+0000 7f621cff9640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:50:46.669 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.669+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f620c004eb0 con 0x7f62200b4d60 2024-03-25T19:50:46.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.677+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411987970 0 0) 0x7f6220138480 con 0x7f62200b1730 2024-03-25T19:50:46.676 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.677+0000 7f621cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.677+0000 7f621cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.677+0000 7f621cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.677+0000 7f621cff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:50:46.677 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.677+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6220141130 con 0x7f62200b1730 2024-03-25T19:50:46.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.681+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564049779 0 0) 0x7f6220143d40 con 0x7f622014be90 2024-03-25T19:50:46.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.682+0000 7f621cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.682+0000 7f621cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.682+0000 7f621cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.682+0000 7f621cff9640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:50:46.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.682+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6220138480 con 0x7f622014be90 2024-03-25T19:50:46.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.692+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 853714787 0 0) 0x7f6220138480 con 0x7f622014be90 2024-03-25T19:50:46.691 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.692+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6220143d40 con 0x7f622014be90 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.692+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6218001a30 con 0x7f622014be90 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.692+0000 7f621cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.692+0000 7f621cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.692+0000 7f621cff9640 10 monclient(hunting): dump: 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:46.692 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.692+0000 7f621cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.693+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3464703045 0 0) 0x7f620c004eb0 con 0x7f62200b4d60 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.693+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6220138480 con 0x7f62200b4d60 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.693+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6214001670 con 0x7f62200b4d60 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.693+0000 7f621cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.693+0000 7f621cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.693+0000 7f621cff9640 10 monclient(hunting): dump: 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.693 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.694 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:46.694 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:46.694 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:46.694 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:46.694 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.702+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657741601 0 0) 0x7f6220141130 con 0x7f62200b1730 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.702+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f620c004eb0 con 0x7f62200b1730 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.702+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6210001870 con 0x7f62200b1730 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.702+0000 7f621cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.702+0000 7f621cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.702+0000 7f621cff9640 10 monclient(hunting): dump: 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:46.702 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.703 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:46.703 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:46.703 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:46.703 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:46.703 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3756752881 0 0) 0x7f6220143d40 con 0x7f622014be90 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 >> v1:172.21.15.110:6790/0 conn(0x7f62200b4d60 legacy=0x7f6220142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 >> v1:172.21.15.178:6789/0 conn(0x7f62200b1730 legacy=0x7f6220137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 1 monclient: found mon.a 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6220137660 con 0x7f622014be90 2024-03-25T19:50:46.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f621cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 5 monclient: authenticate success, global_id 4692 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 10 monclient: _renew_subs 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6220137360 con 0x7f622014be90 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 10 monclient: _renew_subs 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6220143180 con 0x7f622014be90 2024-03-25T19:50:46.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.706+0000 7f6226390640 1 librados: init done 2024-03-25T19:50:46.708 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.707+0000 7f6226390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:46.708 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.707+0000 7f6226390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:46.708 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.707+0000 7f6226390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:46.708 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.707+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6220140f30 con 0x7f622014be90 2024-03-25T19:50:46.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.725+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6218001f20 con 0x7f622014be90 2024-03-25T19:50:46.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.725+0000 7f621cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:46.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:46.725+0000 7f621e7fc640 4 set_mon_vals no callback set 2024-03-25T19:50:47.016 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.017+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6218004d00 con 0x7f622014be90 2024-03-25T19:50:47.016 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.017+0000 7f621cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:47.016 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.017+0000 7f621cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.017+0000 7f621cff9640 10 monclient: dump: 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:47.017 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:47.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.017+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f621801acd0 con 0x7f622014be90 2024-03-25T19:50:47.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.018+0000 7f621ffff640 10 monclient: get_auth_request con 0x7f620c06e0f0 auth_method 0 2024-03-25T19:50:47.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.018+0000 7f621cff9640 10 client.4692.objecter ms_handle_connect 0x7f620c06e0f0 2024-03-25T19:50:47.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.040+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f6218054900 con 0x7f622014be90 2024-03-25T19:50:47.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.040+0000 7f621cff9640 10 client.4692.objecter ms_dispatch 0x7f6220000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:50:47.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.040+0000 7f621cff9640 3 client.4692.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:50:47.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.040+0000 7f621cff9640 3 client.4692.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:50:47.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.040+0000 7f621cff9640 20 client.4692.objecter dump_active .. 0 homeless 2024-03-25T19:50:47.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.040+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6218050240 con 0x7f622014be90 2024-03-25T19:50:47.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.040+0000 7f621cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:47.040 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.040+0000 7f621cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:47.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.251+0000 7f6226390640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:47.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.251+0000 7f6226390640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:47.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.251+0000 7f6226390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:50:47.251 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.251+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62200627b0 con 0x7f622014be90 2024-03-25T19:50:47.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.287+0000 7f621cff9640 1 -- 172.21.15.110:0/428980261 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f6218052340 con 0x7f622014be90 2024-03-25T19:50:47.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.287+0000 7f621cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:47.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.287+0000 7f621cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:47.287 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:50:47.287 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:47.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.289+0000 7f6226390640 10 librados: watch_flush enter 2024-03-25T19:50:47.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.289+0000 7f6226390640 10 librados: watch_flush exit 2024-03-25T19:50:47.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.289+0000 7f6226390640 20 client.4692.objecter shutdown clearing up homeless session... 2024-03-25T19:50:47.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.289+0000 7f6226390640 10 client.4692.objecter successfully canceled tick 2024-03-25T19:50:47.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 >> v1:172.21.15.178:6800/1132739058 conn(0x7f620c06e0f0 legacy=0x7f620c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:47.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 10 monclient: shutdown 2024-03-25T19:50:47.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:47.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 >> v1:172.21.15.110:6789/0 conn(0x7f622014be90 legacy=0x7f62201d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:47.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 shutdown_connections 2024-03-25T19:50:47.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 >> 172.21.15.110:0/428980261 conn(0x7f62200617a0 msgr2=0x7f62200b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:47.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 shutdown_connections 2024-03-25T19:50:47.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 1 -- 172.21.15.110:0/428980261 wait complete. 2024-03-25T19:50:47.291 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:47.290+0000 7f6226390640 1 librados: shutdown 2024-03-25T19:50:47.301 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:54.108 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:54.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.220+0000 7f6ee034d640 10 monclient: get_monmap_and_config 2024-03-25T19:50:54.220 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.220+0000 7f6ee034d640 10 monclient: build_initial_monmap 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.221+0000 7f6ee034d640 10 monclient: monmap: 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:50:54.221948+0000 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:50:54.221948+0000 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:50:54.221 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:54.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.221+0000 7f6ee034d640 1 Processor -- start 2024-03-25T19:50:54.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.221+0000 7f6ee034d640 1 -- start start 2024-03-25T19:50:54.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.221+0000 7f6ee034d640 10 monclient: init 2024-03-25T19:50:54.222 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ed80705b0 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ed81402d0 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ed8149ae0 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed80718c0 con 0x7f6ed80705b0 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed80714c0 con 0x7f6ed81402d0 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed8072b50 con 0x7f6ed8149ae0 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:54.223 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.222+0000 7f6ee034d640 10 monclient(hunting): authenticate will time out at 4825.363695s 2024-03-25T19:50:54.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.239+0000 7f6edd8e8640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f6ed81402d0 0x7f6ed8140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:37344/0 (socket says 172.21.15.110:37344) 2024-03-25T19:50:54.239 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.239+0000 7f6edd8e8640 1 -- 172.21.15.110:0/2073938439 learned_addr learned my addr 172.21.15.110:0/2073938439 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:54.248 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.249+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042934812 0 0) 0x7f6ed80714c0 con 0x7f6ed81402d0 2024-03-25T19:50:54.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.249+0000 7f6edd0e7640 10 monclient(hunting): _negotiate 2024-03-25T19:50:54.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.249+0000 7f6edd0e7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:54.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.249+0000 7f6edd0e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:54.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.249+0000 7f6edd0e7640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:50:54.249 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.249+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec4003520 con 0x7f6ed81402d0 2024-03-25T19:50:54.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.259+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007773974 0 0) 0x7f6ed8072b50 con 0x7f6ed8149ae0 2024-03-25T19:50:54.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.259+0000 7f6edd0e7640 10 monclient(hunting): _negotiate 2024-03-25T19:50:54.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.259+0000 7f6edd0e7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:54.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.259+0000 7f6edd0e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:54.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.259+0000 7f6edd0e7640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:50:54.259 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.259+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed80714c0 con 0x7f6ed8149ae0 2024-03-25T19:50:54.262 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.262+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699917319 0 0) 0x7f6ec4003520 con 0x7f6ed81402d0 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.262+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 >> v1:172.21.15.110:6790/0 conn(0x7f6ed8149ae0 legacy=0x7f6ed814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.262+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 >> v1:172.21.15.110:6789/0 conn(0x7f6ed80705b0 legacy=0x7f6ed813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.262+0000 7f6edd0e7640 10 monclient: _finish_hunting 0 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.262+0000 7f6edd0e7640 1 monclient: found mon.noname-c 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed813dd10 con 0x7f6ed81402d0 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 10 monclient: _finish_auth 0 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 10 monclient: _check_auth_tickets 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6ec8001690 con 0x7f6ed81402d0 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:54.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6edd0e7640 10 monclient: dump: 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6ee034d640 5 monclient: authenticate success, global_id 4702 2024-03-25T19:50:54.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.263+0000 7f6ee034d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:54.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.276+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ec8001870 con 0x7f6ed81402d0 2024-03-25T19:50:54.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.276+0000 7f6edd0e7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:54.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.276+0000 7f6edd0e7640 1 -- 172.21.15.110:0/2073938439 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6ec8001c80 con 0x7f6ed81402d0 2024-03-25T19:50:54.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.276+0000 7f6edd0e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:54.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.276+0000 7f6edd0e7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:54.276 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.276+0000 7f6edd0e7640 10 monclient: dump: 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.276+0000 7f6ee034d640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:54.277 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.276+0000 7f6ee034d640 4 set_mon_vals no callback set 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.277+0000 7f6ee034d640 10 monclient: shutdown 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.277+0000 7f6ee034d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.277+0000 7f6ee034d640 1 -- 172.21.15.110:0/2073938439 >> v1:172.21.15.178:6789/0 conn(0x7f6ed81402d0 legacy=0x7f6ed8140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.277+0000 7f6ee034d640 1 -- 172.21.15.110:0/2073938439 shutdown_connections 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.277+0000 7f6ee034d640 1 -- 172.21.15.110:0/2073938439 >> 172.21.15.110:0/2073938439 conn(0x7f6ed80b5d10 msgr2=0x7f6ed80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.277+0000 7f6ee034d640 1 -- 172.21.15.110:0/2073938439 shutdown_connections 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.277+0000 7f6ee034d640 1 -- 172.21.15.110:0/2073938439 wait complete. 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.278+0000 7f6ee034d640 10 monclient: build_initial_monmap 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.278+0000 7f6ee034d640 10 monclient: monmap: 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:54.279 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.278+0000 7f6ee034d640 1 librados: starting msgr at 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.278+0000 7f6ee034d640 1 librados: starting objecter 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.278+0000 7f6ee034d640 1 Processor -- start 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 1 -- start start 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 1 librados: setting wanted keys 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 1 librados: calling monclient init 2024-03-25T19:50:54.280 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 10 monclient: init 2024-03-25T19:50:54.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:54.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:54.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ed80705b0 addr v1:172.21.15.110:6789/0 2024-03-25T19:50:54.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ed81402d0 addr v1:172.21.15.110:6790/0 2024-03-25T19:50:54.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ed8149ae0 addr v1:172.21.15.178:6789/0 2024-03-25T19:50:54.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed81d56d0 con 0x7f6ed80705b0 2024-03-25T19:50:54.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed8141f20 con 0x7f6ed8149ae0 2024-03-25T19:50:54.281 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed81d6270 con 0x7f6ed81402d0 2024-03-25T19:50:54.282 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:54.282 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.279+0000 7f6ee034d640 10 monclient(hunting): authenticate will time out at 4825.420759s 2024-03-25T19:50:54.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.285+0000 7f6edd8e8640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f6ed81402d0 0x7f6ed8131df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:57474/0 (socket says 172.21.15.110:57474) 2024-03-25T19:50:54.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.285+0000 7f6edd8e8640 1 -- 172.21.15.110:0/1792686610 learned_addr learned my addr 172.21.15.110:0/1792686610 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:50:54.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.287+0000 7f6eceffd640 10 client.?.objecter ms_handle_connect 0x7f6ed81402d0 2024-03-25T19:50:54.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.287+0000 7f6eceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:54.292 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.293+0000 7f6eceffd640 10 client.?.objecter ms_handle_connect 0x7f6ed8149ae0 2024-03-25T19:50:54.292 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.293+0000 7f6eceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:54.294 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.295+0000 7f6eceffd640 10 client.?.objecter ms_handle_connect 0x7f6ed80705b0 2024-03-25T19:50:54.294 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.295+0000 7f6eceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:54.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.296+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117331442 0 0) 0x7f6ed81d6270 con 0x7f6ed81402d0 2024-03-25T19:50:54.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.296+0000 7f6eceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:54.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.296+0000 7f6eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:54.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.296+0000 7f6eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:54.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.296+0000 7f6eceffd640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:50:54.296 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.296+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ebc004eb0 con 0x7f6ed81402d0 2024-03-25T19:50:54.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.302+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004223302 0 0) 0x7f6ed81d56d0 con 0x7f6ed80705b0 2024-03-25T19:50:54.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.302+0000 7f6eceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:54.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.302+0000 7f6eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:54.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.302+0000 7f6eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:54.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.302+0000 7f6eceffd640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:50:54.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.302+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed81d6270 con 0x7f6ed80705b0 2024-03-25T19:50:54.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.304+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689288031 0 0) 0x7f6ed8141f20 con 0x7f6ed8149ae0 2024-03-25T19:50:54.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.304+0000 7f6eceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:54.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.304+0000 7f6eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:54.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.304+0000 7f6eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:54.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.304+0000 7f6eceffd640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:50:54.304 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.304+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed81d56d0 con 0x7f6ed8149ae0 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.308+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 854873096 0 0) 0x7f6ebc004eb0 con 0x7f6ed81402d0 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.308+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ed8141f20 con 0x7f6ed81402d0 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.308+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6ec8001bc0 con 0x7f6ed81402d0 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.308+0000 7f6eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.308+0000 7f6eceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.308+0000 7f6eceffd640 10 monclient(hunting): dump: 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:54.308 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:54.309 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:54.309 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:54.309 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:54.309 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:54.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.308+0000 7f6eceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:54.310 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.310+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114822155 0 0) 0x7f6ed81d6270 con 0x7f6ed80705b0 2024-03-25T19:50:54.310 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.310+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ebc004eb0 con 0x7f6ed80705b0 2024-03-25T19:50:54.310 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.310+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6ed40018a0 con 0x7f6ed80705b0 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.310+0000 7f6eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.310+0000 7f6eceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.310+0000 7f6eceffd640 10 monclient(hunting): dump: 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:54.311 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:54.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 711709038 0 0) 0x7f6ed8141f20 con 0x7f6ed81402d0 2024-03-25T19:50:54.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 >> v1:172.21.15.178:6789/0 conn(0x7f6ed8149ae0 legacy=0x7f6ed813ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 >> v1:172.21.15.110:6789/0 conn(0x7f6ed80705b0 legacy=0x7f6ed81d5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:54.314 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 1 monclient: found mon.c 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed81d6400 con 0x7f6ed81402d0 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6eceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 5 monclient: authenticate success, global_id 4706 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 10 monclient: _renew_subs 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed8136030 con 0x7f6ed81402d0 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 10 client.4706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 10 monclient: _renew_subs 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed81326e0 con 0x7f6ed81402d0 2024-03-25T19:50:54.315 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.314+0000 7f6ee034d640 1 librados: init done 2024-03-25T19:50:54.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.315+0000 7f6ee034d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:54.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.315+0000 7f6ee034d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:54.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.315+0000 7f6ee034d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:50:54.316 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.315+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed81d54a0 con 0x7f6ed81402d0 2024-03-25T19:50:54.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.638+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ec8004bf0 con 0x7f6ed81402d0 2024-03-25T19:50:54.638 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.638+0000 7f6eceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.638+0000 7f6edc8e6640 4 set_mon_vals no callback set 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.639+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6ec8004f00 con 0x7f6ed81402d0 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.639+0000 7f6eceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.639+0000 7f6eceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.639+0000 7f6eceffd640 10 monclient: dump: 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.639 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.639+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f6ec801aed0 con 0x7f6ed81402d0 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.640+0000 7f6ede0e9640 10 monclient: get_auth_request con 0x7f6ebc06df00 auth_method 0 2024-03-25T19:50:54.640 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.640+0000 7f6eceffd640 10 client.4706.objecter ms_handle_connect 0x7f6ebc06df00 2024-03-25T19:50:54.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.647+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f6ec8054a80 con 0x7f6ed81402d0 2024-03-25T19:50:54.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.647+0000 7f6eceffd640 10 client.4706.objecter ms_dispatch 0x7f6ed8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:50:54.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.647+0000 7f6eceffd640 3 client.4706.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:50:54.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.647+0000 7f6eceffd640 3 client.4706.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:50:54.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.647+0000 7f6eceffd640 20 client.4706.objecter dump_active .. 0 homeless 2024-03-25T19:50:54.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.647+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ec8050590 con 0x7f6ed81402d0 2024-03-25T19:50:54.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.647+0000 7f6eceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:54.647 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.647+0000 7f6eceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:54.881 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.882+0000 7f6ee034d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:54.882 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.882+0000 7f6ee034d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:54.882 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.882+0000 7f6ee034d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:50:54.882 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.882+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ed80627b0 con 0x7f6ed81402d0 2024-03-25T19:50:54.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.931+0000 7f6eceffd640 1 -- 172.21.15.110:0/1792686610 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f6ec80524c0 con 0x7f6ed81402d0 2024-03-25T19:50:54.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.931+0000 7f6eceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:54.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.931+0000 7f6eceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:54.931 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:50:54.931 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:54.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 10 librados: watch_flush enter 2024-03-25T19:50:54.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 10 librados: watch_flush exit 2024-03-25T19:50:54.933 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 20 client.4706.objecter shutdown clearing up homeless session... 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 10 client.4706.objecter successfully canceled tick 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 >> v1:172.21.15.178:6800/1132739058 conn(0x7f6ebc06df00 legacy=0x7f6ebc0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 10 monclient: shutdown 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 >> v1:172.21.15.110:6790/0 conn(0x7f6ed81402d0 legacy=0x7f6ed8131df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 shutdown_connections 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 >> 172.21.15.110:0/1792686610 conn(0x7f6ed80b5d10 msgr2=0x7f6ed80b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 shutdown_connections 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 1 -- 172.21.15.110:0/1792686610 wait complete. 2024-03-25T19:50:54.934 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:50:54.934+0000 7f6ee034d640 1 librados: shutdown 2024-03-25T19:50:54.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:01.753 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:01.865 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.865+0000 7f187e115640 10 monclient: get_monmap_and_config 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.865+0000 7f187e115640 10 monclient: build_initial_monmap 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.866+0000 7f187e115640 10 monclient: monmap: 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:51:01.866932+0000 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:51:01.866932+0000 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:51:01.866 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:01.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.867+0000 7f187e115640 1 Processor -- start 2024-03-25T19:51:01.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.867+0000 7f187e115640 1 -- start start 2024-03-25T19:51:01.867 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.867+0000 7f187e115640 10 monclient: init 2024-03-25T19:51:01.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:01.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:01.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 10 monclient(hunting): picked mon.noname-c con 0x7f18780a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:01.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 10 monclient(hunting): picked mon.noname-a con 0x7f187813f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:01.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 10 monclient(hunting): picked mon.noname-b con 0x7f18781430c0 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:01.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1878142520 con 0x7f187813f710 2024-03-25T19:51:01.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780b28a0 con 0x7f18780a98f0 2024-03-25T19:51:01.868 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780b2a30 con 0x7f18781430c0 2024-03-25T19:51:01.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:01.869 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.868+0000 7f187e115640 10 monclient(hunting): authenticate will time out at 4833.009332s 2024-03-25T19:51:01.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.880+0000 7f18777fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f18780a98f0 0x7f18780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:34608/0 (socket says 172.21.15.110:34608) 2024-03-25T19:51:01.880 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.880+0000 7f18777fe640 1 -- 172.21.15.110:0/232713237 learned_addr learned my addr 172.21.15.110:0/232713237 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928653317 0 0) 0x7f1878142520 con 0x7f187813f710 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1860003520 con 0x7f187813f710 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184347978 0 0) 0x7f18780b28a0 con 0x7f18780a98f0 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:51:01.902 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.902+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1878142520 con 0x7f18780a98f0 2024-03-25T19:51:01.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.905+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966019980 0 0) 0x7f18780b2a30 con 0x7f18781430c0 2024-03-25T19:51:01.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.905+0000 7f18767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.905+0000 7f18767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.905+0000 7f18767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.905+0000 7f18767fc640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:51:01.905 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.905+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18780b28a0 con 0x7f18781430c0 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.913+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1587478431 0 0) 0x7f1878142520 con 0x7f18780a98f0 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.913+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 >> v1:172.21.15.110:6790/0 conn(0x7f18781430c0 legacy=0x7f187814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.913+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 >> v1:172.21.15.110:6789/0 conn(0x7f187813f710 legacy=0x7f18781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 1 monclient: found mon.noname-c 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18780b44c0 con 0x7f18780a98f0 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1868001670 con 0x7f18780a98f0 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f18767fc640 10 monclient: dump: 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f187e115640 5 monclient: authenticate success, global_id 4708 2024-03-25T19:51:01.915 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.914+0000 7f187e115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.926+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18680017d0 con 0x7f18780a98f0 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.926+0000 7f18767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.926+0000 7f18767fc640 1 -- 172.21.15.110:0/232713237 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1868001be0 con 0x7f18780a98f0 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.926+0000 7f18767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.926+0000 7f18767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.926+0000 7f18767fc640 10 monclient: dump: 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:01.927 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:01.927 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:01.927 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:01.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.926+0000 7f187e115640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:01.927 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.926+0000 7f187e115640 4 set_mon_vals no callback set 2024-03-25T19:51:01.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.927+0000 7f187e115640 10 monclient: shutdown 2024-03-25T19:51:01.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.927+0000 7f187e115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:01.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.927+0000 7f187e115640 1 -- 172.21.15.110:0/232713237 >> v1:172.21.15.178:6789/0 conn(0x7f18780a98f0 legacy=0x7f18780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.927+0000 7f187e115640 1 -- 172.21.15.110:0/232713237 shutdown_connections 2024-03-25T19:51:01.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.927+0000 7f187e115640 1 -- 172.21.15.110:0/232713237 >> 172.21.15.110:0/232713237 conn(0x7f18780b7ea0 msgr2=0x7f18780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:01.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.927+0000 7f187e115640 1 -- 172.21.15.110:0/232713237 shutdown_connections 2024-03-25T19:51:01.928 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.927+0000 7f187e115640 1 -- 172.21.15.110:0/232713237 wait complete. 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 10 monclient: build_initial_monmap 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 10 monclient: monmap: 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 1 librados: starting msgr at 2024-03-25T19:51:01.929 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 1 librados: starting objecter 2024-03-25T19:51:01.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 1 Processor -- start 2024-03-25T19:51:01.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 1 -- start start 2024-03-25T19:51:01.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 1 librados: setting wanted keys 2024-03-25T19:51:01.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 1 librados: calling monclient init 2024-03-25T19:51:01.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.928+0000 7f187e115640 10 monclient: init 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 10 monclient(hunting): picked mon.noname-c con 0x7f18780a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 10 monclient(hunting): picked mon.noname-b con 0x7f187813f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 10 monclient(hunting): picked mon.noname-a con 0x7f18781430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1878137f50 con 0x7f18781430c0 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1878139830 con 0x7f187813f710 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780aba50 con 0x7f18780a98f0 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:01.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.929+0000 7f187e115640 10 monclient(hunting): authenticate will time out at 4833.070663s 2024-03-25T19:51:01.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.938+0000 7f18777fe640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f18780a98f0 0x7f1878134080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:33592/0 (socket says 172.21.15.110:33592) 2024-03-25T19:51:01.937 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.938+0000 7f18777fe640 1 -- 172.21.15.110:0/3579487610 learned_addr learned my addr 172.21.15.110:0/3579487610 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:01.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.939+0000 7f185ffff640 10 client.?.objecter ms_handle_connect 0x7f187813f710 2024-03-25T19:51:01.939 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.939+0000 7f185ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:01.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.943+0000 7f185ffff640 10 client.?.objecter ms_handle_connect 0x7f18780a98f0 2024-03-25T19:51:01.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.943+0000 7f185ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:01.951 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.952+0000 7f185ffff640 10 client.?.objecter ms_handle_connect 0x7f18781430c0 2024-03-25T19:51:01.952 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.952+0000 7f185ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:01.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.953+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719675436 0 0) 0x7f18780aba50 con 0x7f18780a98f0 2024-03-25T19:51:01.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.953+0000 7f185ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.953+0000 7f185ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.953+0000 7f185ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.953+0000 7f185ffff640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:51:01.953 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.953+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1864004eb0 con 0x7f18780a98f0 2024-03-25T19:51:01.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.955+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134621118 0 0) 0x7f1878139830 con 0x7f187813f710 2024-03-25T19:51:01.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.955+0000 7f185ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.955+0000 7f185ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.955+0000 7f185ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.955+0000 7f185ffff640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:51:01.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.955+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18780aba50 con 0x7f187813f710 2024-03-25T19:51:01.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.966+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536162592 0 0) 0x7f1878137f50 con 0x7f18781430c0 2024-03-25T19:51:01.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.966+0000 7f185ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.966+0000 7f185ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.966+0000 7f185ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.967+0000 7f185ffff640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:51:01.967 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.967+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1878139830 con 0x7f18781430c0 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.971+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931263933 0 0) 0x7f1864004eb0 con 0x7f18780a98f0 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.971+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1878137f50 con 0x7f18780a98f0 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.971+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f1868001c20 con 0x7f18780a98f0 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.971+0000 7f185ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.971+0000 7f185ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.971+0000 7f185ffff640 10 monclient(hunting): dump: 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.971 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:01.972 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:01.972 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:01.972 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:01.972 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:01.972 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.971+0000 7f185ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:01.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.976+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570783109 0 0) 0x7f18780aba50 con 0x7f187813f710 2024-03-25T19:51:01.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.976+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1864004eb0 con 0x7f187813f710 2024-03-25T19:51:01.975 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.976+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f186c001950 con 0x7f187813f710 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.976+0000 7f185ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.976+0000 7f185ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.976+0000 7f185ffff640 10 monclient(hunting): dump: 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:01.976 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:01.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 478494431 0 0) 0x7f1864004eb0 con 0x7f187813f710 2024-03-25T19:51:01.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 >> v1:172.21.15.110:6790/0 conn(0x7f18780a98f0 legacy=0x7f1878134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 >> v1:172.21.15.110:6789/0 conn(0x7f18781430c0 legacy=0x7f1878136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 1 monclient: found mon.b 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18780abd30 con 0x7f187813f710 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 10 monclient: _finish_auth 0 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f185ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 10 monclient: _renew_subs 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1878137a00 con 0x7f187813f710 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 10 monclient: _renew_subs 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:01.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1878134970 con 0x7f187813f710 2024-03-25T19:51:01.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.985+0000 7f187e115640 1 librados: init done 2024-03-25T19:51:01.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.986+0000 7f187e115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:01.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.986+0000 7f187e115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:01.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.986+0000 7f187e115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:01.987 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.986+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18780627b0 con 0x7f187813f710 2024-03-25T19:51:01.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.994+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f186c001d60 con 0x7f187813f710 2024-03-25T19:51:01.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.994+0000 7f185ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:01.994 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:01.994+0000 7f1875ffb640 4 set_mon_vals no callback set 2024-03-25T19:51:02.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.283+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f186c003790 con 0x7f187813f710 2024-03-25T19:51:02.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.283+0000 7f185ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:02.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.283+0000 7f185ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:02.283 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.283+0000 7f185ffff640 10 monclient: dump: 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:02.284 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:02.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.284+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f186c0038f0 con 0x7f187813f710 2024-03-25T19:51:02.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.285+0000 7f18777fe640 10 monclient: get_auth_request con 0x7f186406df10 auth_method 0 2024-03-25T19:51:02.285 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.285+0000 7f185ffff640 10 client.4711.objecter ms_handle_connect 0x7f186406df10 2024-03-25T19:51:02.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.288+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f186c0535e0 con 0x7f187813f710 2024-03-25T19:51:02.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.288+0000 7f185ffff640 10 client.4711.objecter ms_dispatch 0x7f1878000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:02.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.288+0000 7f185ffff640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:02.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.288+0000 7f185ffff640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:02.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.288+0000 7f185ffff640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T19:51:02.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.291+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f186c082e60 con 0x7f187813f710 2024-03-25T19:51:02.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.291+0000 7f185ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:02.290 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.291+0000 7f185ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:02.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.542+0000 7f187e115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:02.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.542+0000 7f187e115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:02.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.542+0000 7f187e115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:02.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.542+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18780b71a0 con 0x7f187813f710 2024-03-25T19:51:02.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.573+0000 7f185ffff640 1 -- 172.21.15.110:0/3579487610 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f186c052cb0 con 0x7f187813f710 2024-03-25T19:51:02.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.573+0000 7f185ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:02.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.573+0000 7f185ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:02.574 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:51:02.574 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:02.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 10 librados: watch_flush enter 2024-03-25T19:51:02.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 10 librados: watch_flush exit 2024-03-25T19:51:02.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 10 client.4711.objecter successfully canceled tick 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 >> v1:172.21.15.178:6800/1132739058 conn(0x7f186406df10 legacy=0x7f18640703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 10 monclient: shutdown 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 >> v1:172.21.15.178:6789/0 conn(0x7f187813f710 legacy=0x7f1878135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 shutdown_connections 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.576+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 >> 172.21.15.110:0/3579487610 conn(0x7f18780b7ea0 msgr2=0x7f1878074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.577+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 shutdown_connections 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.577+0000 7f187e115640 1 -- 172.21.15.110:0/3579487610 wait complete. 2024-03-25T19:51:02.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:02.577+0000 7f187e115640 1 librados: shutdown 2024-03-25T19:51:02.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:04.218 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:11.502 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:04.263 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:11.548 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:04.276 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:11.530 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:04.285 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:11.569 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:04.285 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:11.538 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:04.312 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:11.596 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:04.324 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:11.579 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:04.329 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:11.613 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:04.353 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:11.637 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:09.396 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.505+0000 7f8c0fb4b640 10 monclient: get_monmap_and_config 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.505+0000 7f8c0fb4b640 10 monclient: build_initial_monmap 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.505+0000 7f8c0fb4b640 10 monclient: monmap: 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:51:09.506398+0000 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:51:09.506398+0000 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:09.505 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:09.506 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:09.506 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:51:09.506 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:51:09.506 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:09.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.506+0000 7f8c0fb4b640 1 Processor -- start 2024-03-25T19:51:09.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.506+0000 7f8c0fb4b640 1 -- start start 2024-03-25T19:51:09.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.506+0000 7f8c0fb4b640 10 monclient: init 2024-03-25T19:51:09.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:09.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:09.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f8c080b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:09.507 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f8c080b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:09.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f8c0814be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:09.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c080af940 con 0x7f8c0814be90 2024-03-25T19:51:09.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c080b0a70 con 0x7f8c080b1730 2024-03-25T19:51:09.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c080b0d50 con 0x7f8c080b4d60 2024-03-25T19:51:09.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:09.508 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.507+0000 7f8c0fb4b640 10 monclient(hunting): authenticate will time out at 4840.648526s 2024-03-25T19:51:09.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.517+0000 7f8c0d8e7640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f8c080b4d60 0x7f8c0813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:38292/0 (socket says 172.21.15.110:38292) 2024-03-25T19:51:09.517 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.517+0000 7f8c0d8e7640 1 -- 172.21.15.110:0/139483247 learned_addr learned my addr 172.21.15.110:0/139483247 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:09.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.527+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96077405 0 0) 0x7f8c080b0a70 con 0x7f8c080b1730 2024-03-25T19:51:09.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.527+0000 7f8c0c8e5640 10 monclient(hunting): _negotiate 2024-03-25T19:51:09.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.527+0000 7f8c0c8e5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:09.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.527+0000 7f8c0c8e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:09.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.527+0000 7f8c0c8e5640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:51:09.527 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.527+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf4003520 con 0x7f8c080b1730 2024-03-25T19:51:09.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.542+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694927680 0 0) 0x7f8c080b0d50 con 0x7f8c080b4d60 2024-03-25T19:51:09.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.542+0000 7f8c0c8e5640 10 monclient(hunting): _negotiate 2024-03-25T19:51:09.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.542+0000 7f8c0c8e5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:09.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.542+0000 7f8c0c8e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:09.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.542+0000 7f8c0c8e5640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:51:09.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.542+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c080b0a70 con 0x7f8c080b4d60 2024-03-25T19:51:09.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776973646 0 0) 0x7f8bf4003520 con 0x7f8c080b1730 2024-03-25T19:51:09.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 >> v1:172.21.15.110:6790/0 conn(0x7f8c080b4d60 legacy=0x7f8c0813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:09.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 >> v1:172.21.15.110:6789/0 conn(0x7f8c0814be90 legacy=0x7f8c0814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:09.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 10 monclient: _finish_hunting 0 2024-03-25T19:51:09.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 1 monclient: found mon.noname-c 2024-03-25T19:51:09.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:09.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:51:09.558 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c080b1030 con 0x7f8c080b1730 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 10 monclient: _finish_auth 0 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 10 monclient: _check_auth_tickets 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8bf8001690 con 0x7f8c080b1730 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0fb4b640 5 monclient: authenticate success, global_id 4714 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0fb4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0c8e5640 10 monclient: dump: 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:09.559 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:09.560 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:09.560 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:09.560 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:09.560 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:09.560 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:09.560 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.558+0000 7f8c0fb4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:09.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.573+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bf80017f0 con 0x7f8c080b1730 2024-03-25T19:51:09.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.573+0000 7f8c0c8e5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:09.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.573+0000 7f8c0c8e5640 1 -- 172.21.15.110:0/139483247 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8bf8001c00 con 0x7f8c080b1730 2024-03-25T19:51:09.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.573+0000 7f8c0fb4b640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:09.573 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.573+0000 7f8c0fb4b640 4 set_mon_vals no callback set 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.573+0000 7f8c0c8e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.573+0000 7f8c0c8e5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.573+0000 7f8c0c8e5640 10 monclient: dump: 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:09.574 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:09.575 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.574+0000 7f8c0fb4b640 10 monclient: shutdown 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.574+0000 7f8c0fb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.574+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/139483247 >> v1:172.21.15.178:6789/0 conn(0x7f8c080b1730 legacy=0x7f8c080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.574+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/139483247 shutdown_connections 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.574+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/139483247 >> 172.21.15.110:0/139483247 conn(0x7f8c080617a0 msgr2=0x7f8c08061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.574+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/139483247 shutdown_connections 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.574+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/139483247 wait complete. 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.575+0000 7f8c0fb4b640 10 monclient: build_initial_monmap 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.575+0000 7f8c0fb4b640 10 monclient: monmap: 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:51:09.576 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 1 librados: starting msgr at 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 1 librados: starting objecter 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 1 Processor -- start 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 1 -- start start 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 1 librados: setting wanted keys 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 1 librados: calling monclient init 2024-03-25T19:51:09.577 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 10 monclient: init 2024-03-25T19:51:09.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:09.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.576+0000 7f8c0fb4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:09.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.577+0000 7f8c0fb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f8c080b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:09.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.577+0000 7f8c0fb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f8c080b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:09.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.577+0000 7f8c0fb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f8c0814be90 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:09.578 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.577+0000 7f8c0fb4b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0813acc0 con 0x7f8c080b1730 2024-03-25T19:51:09.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.577+0000 7f8c0fb4b640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c081d8620 con 0x7f8c0814be90 2024-03-25T19:51:09.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.577+0000 7f8c0fb4b640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0813af30 con 0x7f8c080b4d60 2024-03-25T19:51:09.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.577+0000 7f8c0fb4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:09.579 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.577+0000 7f8c0fb4b640 10 monclient(hunting): authenticate will time out at 4840.717969s 2024-03-25T19:51:09.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.583+0000 7f8c0d0e6640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f8c080b4d60 0x7f8c0813bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:38362/0 (socket says 172.21.15.110:38362) 2024-03-25T19:51:09.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.583+0000 7f8c0d0e6640 1 -- 172.21.15.110:0/74386867 learned_addr learned my addr 172.21.15.110:0/74386867 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:09.583 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.584+0000 7f8bfe7fc640 10 client.?.objecter ms_handle_connect 0x7f8c080b4d60 2024-03-25T19:51:09.584 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.584+0000 7f8bfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:09.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.590+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133952658 0 0) 0x7f8c0813af30 con 0x7f8c080b4d60 2024-03-25T19:51:09.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.590+0000 7f8bfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:09.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.590+0000 7f8bfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:09.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.590+0000 7f8bfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:09.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.590+0000 7f8bfe7fc640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:51:09.590 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.590+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bec004eb0 con 0x7f8c080b4d60 2024-03-25T19:51:09.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.600+0000 7f8bfe7fc640 10 client.?.objecter ms_handle_connect 0x7f8c0814be90 2024-03-25T19:51:09.600 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.600+0000 7f8bfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:09.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.609+0000 7f8bfe7fc640 10 client.?.objecter ms_handle_connect 0x7f8c080b1730 2024-03-25T19:51:09.609 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.609+0000 7f8bfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:09.661 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.662+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736883484 0 0) 0x7f8c0813acc0 con 0x7f8c080b1730 2024-03-25T19:51:09.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.662+0000 7f8bfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:09.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.662+0000 7f8bfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:09.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.662+0000 7f8bfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:09.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.662+0000 7f8bfe7fc640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:51:09.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.662+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0813af30 con 0x7f8c080b1730 2024-03-25T19:51:09.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.696+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2671041487 0 0) 0x7f8c0813af30 con 0x7f8c080b1730 2024-03-25T19:51:09.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.696+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c0813acc0 con 0x7f8c080b1730 2024-03-25T19:51:09.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.696+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8c04001cb0 con 0x7f8c080b1730 2024-03-25T19:51:09.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.696+0000 7f8bfe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:09.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.696+0000 7f8bfe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:09.696 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.696+0000 7f8bfe7fc640 10 monclient(hunting): dump: 2024-03-25T19:51:09.696 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:09.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.696+0000 7f8bfe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:09.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.714+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943495310 0 0) 0x7f8c081d8620 con 0x7f8c0814be90 2024-03-25T19:51:09.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.714+0000 7f8bfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:09.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.714+0000 7f8bfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:09.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.714+0000 7f8bfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:09.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.714+0000 7f8bfe7fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:51:09.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.714+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0813af30 con 0x7f8c0814be90 2024-03-25T19:51:09.723 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46470266 0 0) 0x7f8c0813acc0 con 0x7f8c080b1730 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 >> v1:172.21.15.110:6790/0 conn(0x7f8c080b4d60 legacy=0x7f8c0813bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 >> v1:172.21.15.178:6789/0 conn(0x7f8c0814be90 legacy=0x7f8c081e30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 1 monclient: found mon.a 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c081d79f0 con 0x7f8c080b1730 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8bfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8c0fb4b640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8c0fb4b640 10 monclient: _renew_subs 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8c0fb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:09.724 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.723+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0813c310 con 0x7f8c080b1730 2024-03-25T19:51:09.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.724+0000 7f8c0fb4b640 10 client.4710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:09.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.724+0000 7f8c0fb4b640 10 monclient: _renew_subs 2024-03-25T19:51:09.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.724+0000 7f8c0fb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:09.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.724+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c081394e0 con 0x7f8c080b1730 2024-03-25T19:51:09.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.724+0000 7f8c0fb4b640 1 librados: init done 2024-03-25T19:51:09.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.725+0000 7f8c0fb4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:09.725 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.725+0000 7f8c0fb4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:09.726 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.725+0000 7f8c0fb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:09.726 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:09.725+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c081401f0 con 0x7f8c080b1730 2024-03-25T19:51:10.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.114+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c040020c0 con 0x7f8c080b1730 2024-03-25T19:51:10.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.114+0000 7f8bfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:10.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.114+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f8c04004d00 con 0x7f8c080b1730 2024-03-25T19:51:10.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.114+0000 7f8bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:10.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.114+0000 7f8bfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:10.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.114+0000 7f8bfe7fc640 10 monclient: dump: 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:10.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.114+0000 7f8bfffff640 4 set_mon_vals no callback set 2024-03-25T19:51:10.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.115+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f8c04004f20 con 0x7f8c080b1730 2024-03-25T19:51:10.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.115+0000 7f8c0d0e6640 10 monclient: get_auth_request con 0x7f8bec06de10 auth_method 0 2024-03-25T19:51:10.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.116+0000 7f8bfe7fc640 10 client.4710.objecter ms_handle_connect 0x7f8bec06de10 2024-03-25T19:51:10.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.120+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f8c04054b00 con 0x7f8c080b1730 2024-03-25T19:51:10.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.120+0000 7f8bfe7fc640 10 client.4710.objecter ms_dispatch 0x7f8c08000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:10.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.120+0000 7f8bfe7fc640 3 client.4710.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:10.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.120+0000 7f8bfe7fc640 3 client.4710.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:10.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.120+0000 7f8bfe7fc640 20 client.4710.objecter dump_active .. 0 homeless 2024-03-25T19:51:10.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.120+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c04050440 con 0x7f8c080b1730 2024-03-25T19:51:10.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.120+0000 7f8bfe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:10.121 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.120+0000 7f8bfe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:10.369 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.369+0000 7f8c0fb4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:10.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.369+0000 7f8c0fb4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:10.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.369+0000 7f8c0fb4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:10.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.369+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c0813c660 con 0x7f8c080b1730 2024-03-25T19:51:10.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.412+0000 7f8bfe7fc640 1 -- 172.21.15.110:0/74386867 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f8c04052540 con 0x7f8c080b1730 2024-03-25T19:51:10.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.412+0000 7f8bfe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:10.412 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.412+0000 7f8bfe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:10.412 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:51:10.412 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:10.414 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 10 librados: watch_flush enter 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 10 librados: watch_flush exit 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 20 client.4710.objecter shutdown clearing up homeless session... 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 10 client.4710.objecter successfully canceled tick 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 >> v1:172.21.15.178:6800/1132739058 conn(0x7f8bec06de10 legacy=0x7f8bec0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 10 monclient: shutdown 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 >> v1:172.21.15.110:6789/0 conn(0x7f8c080b1730 legacy=0x7f8c081d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 shutdown_connections 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.414+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 >> 172.21.15.110:0/74386867 conn(0x7f8c080617a0 msgr2=0x7f8c080b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.415+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 shutdown_connections 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.415+0000 7f8c0fb4b640 1 -- 172.21.15.110:0/74386867 wait complete. 2024-03-25T19:51:10.415 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:10.415+0000 7f8c0fb4b640 1 librados: shutdown 2024-03-25T19:51:10.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:17.233 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:17.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.348+0000 7f2036a06640 10 monclient: get_monmap_and_config 2024-03-25T19:51:17.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.348+0000 7f2036a06640 10 monclient: build_initial_monmap 2024-03-25T19:51:17.348 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.348+0000 7f2036a06640 10 monclient: monmap: 2024-03-25T19:51:17.348 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:17.348 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:17.348 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:51:17.349430+0000 2024-03-25T19:51:17.348 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:51:17.349430+0000 2024-03-25T19:51:17.348 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:17.349 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:17.349 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:17.349 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:51:17.349 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:51:17.349 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.349+0000 7f2036a06640 1 Processor -- start 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.349+0000 7f2036a06640 1 -- start start 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.349+0000 7f2036a06640 10 monclient: init 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 10 monclient(hunting): picked mon.noname-a con 0x7f20300a98f0 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 10 monclient(hunting): picked mon.noname-b con 0x7f203013f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 10 monclient(hunting): picked mon.noname-c con 0x7f20301430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:17.350 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2030142520 con 0x7f20300a98f0 2024-03-25T19:51:17.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20300b28a0 con 0x7f20301430c0 2024-03-25T19:51:17.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20300b2a30 con 0x7f203013f710 2024-03-25T19:51:17.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:17.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.350+0000 7f2036a06640 10 monclient(hunting): authenticate will time out at 4848.491243s 2024-03-25T19:51:17.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.351+0000 7f202ffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f20300a98f0 0x7f20300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36890/0 (socket says 172.21.15.110:36890) 2024-03-25T19:51:17.351 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.351+0000 7f202ffff640 1 -- 172.21.15.110:0/4117215707 learned_addr learned my addr 172.21.15.110:0/4117215707 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:17.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.359+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579392585 0 0) 0x7f2030142520 con 0x7f20300a98f0 2024-03-25T19:51:17.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.359+0000 7f202effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:17.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.359+0000 7f202effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:17.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.359+0000 7f202effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:17.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.359+0000 7f202effd640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:51:17.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.359+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2018003520 con 0x7f20300a98f0 2024-03-25T19:51:17.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.368+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868045314 0 0) 0x7f20300b2a30 con 0x7f203013f710 2024-03-25T19:51:17.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.368+0000 7f202effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:17.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.368+0000 7f202effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:17.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.368+0000 7f202effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:17.368 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.368+0000 7f202effd640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:51:17.369 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.368+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030142520 con 0x7f203013f710 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3278454341 0 0) 0x7f2018003520 con 0x7f20300a98f0 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 >> v1:172.21.15.110:6790/0 conn(0x7f203013f710 legacy=0x7f20301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 >> v1:172.21.15.178:6789/0 conn(0x7f20301430c0 legacy=0x7f203014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 1 monclient: found mon.noname-a 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20300b44c0 con 0x7f20300a98f0 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 10 monclient: _finish_auth 0 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2020001690 con 0x7f20300a98f0 2024-03-25T19:51:17.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f202effd640 10 monclient: dump: 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f2036a06640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:51:17.372 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.371+0000 7f2036a06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:17.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.381+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20200017f0 con 0x7f20300a98f0 2024-03-25T19:51:17.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.381+0000 7f202effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:17.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.381+0000 7f202effd640 1 -- 172.21.15.110:0/4117215707 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2020001c00 con 0x7f20300a98f0 2024-03-25T19:51:17.381 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 4 set_mon_vals no callback set 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f202effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f202effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f202effd640 10 monclient: dump: 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:17.382 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:17.383 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:17.383 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:17.383 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:17.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 10 monclient: shutdown 2024-03-25T19:51:17.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:17.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 1 -- 172.21.15.110:0/4117215707 >> v1:172.21.15.110:6789/0 conn(0x7f20300a98f0 legacy=0x7f20300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:17.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 1 -- 172.21.15.110:0/4117215707 shutdown_connections 2024-03-25T19:51:17.385 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 1 -- 172.21.15.110:0/4117215707 >> 172.21.15.110:0/4117215707 conn(0x7f20300b7ea0 msgr2=0x7f20300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 1 -- 172.21.15.110:0/4117215707 shutdown_connections 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.382+0000 7f2036a06640 1 -- 172.21.15.110:0/4117215707 wait complete. 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.383+0000 7f2036a06640 10 monclient: build_initial_monmap 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.383+0000 7f2036a06640 10 monclient: monmap: 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:51:17.386 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.384+0000 7f2036a06640 1 librados: starting msgr at 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.384+0000 7f2036a06640 1 librados: starting objecter 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.384+0000 7f2036a06640 1 Processor -- start 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.384+0000 7f2036a06640 1 -- start start 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.384+0000 7f2036a06640 1 librados: setting wanted keys 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.384+0000 7f2036a06640 1 librados: calling monclient init 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.384+0000 7f2036a06640 10 monclient: init 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 10 monclient(hunting): picked mon.noname-c con 0x7f20300a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 10 monclient(hunting): picked mon.noname-b con 0x7f203013f710 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 10 monclient(hunting): picked mon.noname-a con 0x7f20301430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:17.387 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2030138970 con 0x7f20301430c0 2024-03-25T19:51:17.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20301d7810 con 0x7f203013f710 2024-03-25T19:51:17.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2030139990 con 0x7f20300a98f0 2024-03-25T19:51:17.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:17.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.385+0000 7f2036a06640 10 monclient(hunting): authenticate will time out at 4848.526157s 2024-03-25T19:51:17.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.388+0000 7f202f7fe640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f203013f710 0x7f20301d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:34468/0 (socket says 172.21.15.110:34468) 2024-03-25T19:51:17.388 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.388+0000 7f202f7fe640 1 -- 172.21.15.110:0/3890736219 learned_addr learned my addr 172.21.15.110:0/3890736219 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:17.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.391+0000 7f202cff9640 10 client.?.objecter ms_handle_connect 0x7f20300a98f0 2024-03-25T19:51:17.391 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.392+0000 7f202cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:17.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.392+0000 7f202cff9640 10 client.?.objecter ms_handle_connect 0x7f203013f710 2024-03-25T19:51:17.392 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.392+0000 7f202cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:17.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.400+0000 7f202cff9640 10 client.?.objecter ms_handle_connect 0x7f20301430c0 2024-03-25T19:51:17.400 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.400+0000 7f202cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:17.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.406+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101577680 0 0) 0x7f20301d7810 con 0x7f203013f710 2024-03-25T19:51:17.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.406+0000 7f202cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:17.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.406+0000 7f202cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:17.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.406+0000 7f202cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:17.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.406+0000 7f202cff9640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:51:17.406 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.406+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f201c004eb0 con 0x7f203013f710 2024-03-25T19:51:17.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.413+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755701711 0 0) 0x7f2030139990 con 0x7f20300a98f0 2024-03-25T19:51:17.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.413+0000 7f202cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:17.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.413+0000 7f202cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:17.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.413+0000 7f202cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:17.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.413+0000 7f202cff9640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:51:17.413 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.413+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20301d7810 con 0x7f20300a98f0 2024-03-25T19:51:17.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.422+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 558916353 0 0) 0x7f201c004eb0 con 0x7f203013f710 2024-03-25T19:51:17.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.422+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2030139990 con 0x7f203013f710 2024-03-25T19:51:17.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.422+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2024001a10 con 0x7f203013f710 2024-03-25T19:51:17.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.422+0000 7f202cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:17.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.422+0000 7f202cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:17.422 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.422+0000 7f202cff9640 10 monclient(hunting): dump: 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:17.423 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.422+0000 7f202cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:17.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.427+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496519481 0 0) 0x7f2030138970 con 0x7f20301430c0 2024-03-25T19:51:17.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.427+0000 7f202cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:17.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.427+0000 7f202cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:17.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.427+0000 7f202cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:17.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.427+0000 7f202cff9640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:51:17.427 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.427+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f201c004eb0 con 0x7f20301430c0 2024-03-25T19:51:17.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.434+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3189566380 0 0) 0x7f20301d7810 con 0x7f20300a98f0 2024-03-25T19:51:17.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.434+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2030138970 con 0x7f20300a98f0 2024-03-25T19:51:17.433 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.434+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2020001b00 con 0x7f20300a98f0 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.434+0000 7f202cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.434+0000 7f202cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.434+0000 7f202cff9640 10 monclient(hunting): dump: 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:17.434 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1026390344 0 0) 0x7f2030139990 con 0x7f203013f710 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 >> v1:172.21.15.110:6790/0 conn(0x7f20300a98f0 legacy=0x7f2030138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 >> v1:172.21.15.110:6789/0 conn(0x7f20301430c0 legacy=0x7f20301d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 1 monclient: found mon.b 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20301d8440 con 0x7f203013f710 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f202cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f2036a06640 5 monclient: authenticate success, global_id 4723 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f2036a06640 10 monclient: _renew_subs 2024-03-25T19:51:17.440 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f2036a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:17.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.439+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20301d41e0 con 0x7f203013f710 2024-03-25T19:51:17.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.440+0000 7f2036a06640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:17.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.440+0000 7f2036a06640 10 monclient: _renew_subs 2024-03-25T19:51:17.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.440+0000 7f2036a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:17.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.440+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2030138d80 con 0x7f203013f710 2024-03-25T19:51:17.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.440+0000 7f2036a06640 1 librados: init done 2024-03-25T19:51:17.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.441+0000 7f2036a06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:17.441 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.441+0000 7f2036a06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:17.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.441+0000 7f2036a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:17.442 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.441+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20301d79a0 con 0x7f203013f710 2024-03-25T19:51:17.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.663+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2024001c50 con 0x7f203013f710 2024-03-25T19:51:17.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.663+0000 7f202cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:17.663 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.663+0000 7f202e7fc640 4 set_mon_vals no callback set 2024-03-25T19:51:17.715 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.716+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f2024003a60 con 0x7f203013f710 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.716+0000 7f202cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.716+0000 7f202cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.716+0000 7f202cff9640 10 monclient: dump: 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:17.716 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:17.717 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.717+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f2024003c40 con 0x7f203013f710 2024-03-25T19:51:17.717 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.718+0000 7f202ffff640 10 monclient: get_auth_request con 0x7f201c06dfb0 auth_method 0 2024-03-25T19:51:17.717 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.718+0000 7f202cff9640 10 client.4723.objecter ms_handle_connect 0x7f201c06dfb0 2024-03-25T19:51:17.722 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.722+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f20240548e0 con 0x7f203013f710 2024-03-25T19:51:17.722 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.722+0000 7f202cff9640 10 client.4723.objecter ms_dispatch 0x7f2030000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:17.722 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.722+0000 7f202cff9640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:17.722 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.722+0000 7f202cff9640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:17.722 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.722+0000 7f202cff9640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T19:51:17.728 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.728+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20240503f0 con 0x7f203013f710 2024-03-25T19:51:17.728 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.729+0000 7f202cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:17.728 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.729+0000 7f202cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:17.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.966+0000 7f2036a06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:17.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.966+0000 7f2036a06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:17.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.966+0000 7f2036a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:17.966 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:17.966+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20300627b0 con 0x7f203013f710 2024-03-25T19:51:18.075 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.075+0000 7f202cff9640 1 -- 172.21.15.110:0/3890736219 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f2024052320 con 0x7f203013f710 2024-03-25T19:51:18.075 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.075+0000 7f202cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:18.075 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.075+0000 7f202cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:18.075 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:51:18.075 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:18.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 10 librados: watch_flush enter 2024-03-25T19:51:18.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 10 librados: watch_flush exit 2024-03-25T19:51:18.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T19:51:18.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 10 client.4723.objecter successfully canceled tick 2024-03-25T19:51:18.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 >> v1:172.21.15.178:6800/1132739058 conn(0x7f201c06dfb0 legacy=0x7f201c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:18.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 10 monclient: shutdown 2024-03-25T19:51:18.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:18.078 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 >> v1:172.21.15.178:6789/0 conn(0x7f203013f710 legacy=0x7f20301d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:18.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 shutdown_connections 2024-03-25T19:51:18.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 >> 172.21.15.110:0/3890736219 conn(0x7f20300b7ea0 msgr2=0x7f203013c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:18.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 shutdown_connections 2024-03-25T19:51:18.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 1 -- 172.21.15.110:0/3890736219 wait complete. 2024-03-25T19:51:18.079 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:18.078+0000 7f2036a06640 1 librados: shutdown 2024-03-25T19:51:18.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:24.297 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:31.581 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.370 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:31.655 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.409 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:31.693 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.427 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:31.682 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.432 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:31.716 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.435 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:31.690 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.441 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:31.725 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.459 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:31.713 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.473 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:31.757 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.712 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:31.967 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.717 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:31.971 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.745 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:32.000 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.747 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:32.032 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.749 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:32.033 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.768 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:32.052 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.838 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:32.122 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.842 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:32.126 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.882 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:32.166 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.896 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:25.001 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.001+0000 7f952a39f640 10 monclient: get_monmap_and_config 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.001+0000 7f952a39f640 10 monclient: build_initial_monmap 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.001+0000 7f952a39f640 10 monclient: monmap: 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:51:25.003209+0000 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:51:25.003209+0000 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:51:25.002 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:51:25.003 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.002+0000 7f952a39f640 1 Processor -- start 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.002+0000 7f952a39f640 1 -- start start 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.002+0000 7f952a39f640 10 monclient: init 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 10 monclient(hunting): picked mon.noname-b con 0x7f95240af230 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 10 monclient(hunting): picked mon.noname-a con 0x7f95240af610 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 10 monclient(hunting): picked mon.noname-c con 0x7f9524146680 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:25.004 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240ad9b0 con 0x7f95240af610 2024-03-25T19:51:25.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240ac550 con 0x7f9524146680 2024-03-25T19:51:25.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240ab730 con 0x7f95240af230 2024-03-25T19:51:25.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:25.005 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.003+0000 7f952a39f640 10 monclient(hunting): authenticate will time out at 4856.145191s 2024-03-25T19:51:25.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.017+0000 7f9529b9e640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f9524146680 0x7f9524146e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:44262/0 (socket says 172.21.15.110:44262) 2024-03-25T19:51:25.018 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.017+0000 7f9529b9e640 1 -- 172.21.15.110:0/2049053309 learned_addr learned my addr 172.21.15.110:0/2049053309 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:25.027 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.027+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819801319 0 0) 0x7f95240ac550 con 0x7f9524146680 2024-03-25T19:51:25.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.027+0000 7f9517fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:25.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.027+0000 7f9517fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:25.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.027+0000 7f9517fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:25.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.027+0000 7f9517fff640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:51:25.028 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.027+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9510003520 con 0x7f9524146680 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022138740 0 0) 0x7f9510003520 con 0x7f9524146680 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 >> v1:172.21.15.110:6790/0 conn(0x7f95240af230 legacy=0x7f95240ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 >> v1:172.21.15.110:6789/0 conn(0x7f95240af610 legacy=0x7f95240ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 1 monclient: found mon.noname-c 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95240acd80 con 0x7f9524146680 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 10 monclient: _finish_auth 0 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f9517fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:25.043 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f952a39f640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:51:25.044 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.042+0000 7f952a39f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:25.049 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.049+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f951c001690 con 0x7f9524146680 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.049+0000 7f9517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.049+0000 7f9517fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.049+0000 7f9517fff640 10 monclient: dump: 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:25.050 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.049+0000 7f952a39f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f951c001920 con 0x7f9524146680 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f9517fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f9517fff640 1 -- 172.21.15.110:0/2049053309 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f951c001cb0 con 0x7f9524146680 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f9517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f9517fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f9517fff640 10 monclient: dump: 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:25.063 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:25.064 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:25.064 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f952a39f640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:25.064 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f952a39f640 4 set_mon_vals no callback set 2024-03-25T19:51:25.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f952a39f640 10 monclient: shutdown 2024-03-25T19:51:25.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f952a39f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:25.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.062+0000 7f952a39f640 1 -- 172.21.15.110:0/2049053309 >> v1:172.21.15.178:6789/0 conn(0x7f9524146680 legacy=0x7f9524146e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:25.065 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.063+0000 7f952a39f640 1 -- 172.21.15.110:0/2049053309 shutdown_connections 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.063+0000 7f952a39f640 1 -- 172.21.15.110:0/2049053309 >> 172.21.15.110:0/2049053309 conn(0x7f95240b0320 msgr2=0x7f95240b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.063+0000 7f952a39f640 1 -- 172.21.15.110:0/2049053309 shutdown_connections 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.063+0000 7f952a39f640 1 -- 172.21.15.110:0/2049053309 wait complete. 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.063+0000 7f952a39f640 10 monclient: build_initial_monmap 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.063+0000 7f952a39f640 10 monclient: monmap: 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:25.066 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 librados: starting msgr at 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 librados: starting objecter 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 Processor -- start 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 -- start start 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 librados: setting wanted keys 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 librados: calling monclient init 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 10 monclient: init 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 10 monclient(hunting): picked mon.noname-c con 0x7f95240af230 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 10 monclient(hunting): picked mon.noname-a con 0x7f9524146680 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 10 monclient(hunting): picked mon.noname-b con 0x7f952406fc40 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240b00f0 con 0x7f9524146680 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952406edc0 con 0x7f952406fc40 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952406f0a0 con 0x7f95240af230 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:25.067 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.064+0000 7f952a39f640 10 monclient(hunting): authenticate will time out at 4856.206700s 2024-03-25T19:51:25.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.085+0000 7f9529b9e640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f952406fc40 0x7f9524140020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:44298/0 (socket says 172.21.15.110:44298) 2024-03-25T19:51:25.086 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.085+0000 7f9529b9e640 1 -- 172.21.15.110:0/1620674509 learned_addr learned my addr 172.21.15.110:0/1620674509 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:25.088 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.087+0000 7f9515ffb640 10 client.?.objecter ms_handle_connect 0x7f952406fc40 2024-03-25T19:51:25.088 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.087+0000 7f9515ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:25.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.097+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060579364 0 0) 0x7f952406edc0 con 0x7f952406fc40 2024-03-25T19:51:25.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.097+0000 7f9515ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:25.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.097+0000 7f9515ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:25.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.097+0000 7f9515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:25.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.097+0000 7f9515ffb640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:51:25.098 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.097+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9508004eb0 con 0x7f952406fc40 2024-03-25T19:51:25.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.100+0000 7f9515ffb640 10 client.?.objecter ms_handle_connect 0x7f95240af230 2024-03-25T19:51:25.101 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.100+0000 7f9515ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:25.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.102+0000 7f9515ffb640 10 client.?.objecter ms_handle_connect 0x7f9524146680 2024-03-25T19:51:25.103 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.102+0000 7f9515ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:25.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.108+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862406943 0 0) 0x7f95240b00f0 con 0x7f9524146680 2024-03-25T19:51:25.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.108+0000 7f9515ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:25.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.108+0000 7f9515ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:25.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.108+0000 7f9515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:25.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.108+0000 7f9515ffb640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:51:25.109 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.108+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f952406edc0 con 0x7f9524146680 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.113+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1359070480 0 0) 0x7f9508004eb0 con 0x7f952406fc40 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.113+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.178:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95240b00f0 con 0x7f952406fc40 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.113+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f951c001650 con 0x7f952406fc40 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.113+0000 7f9515ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.113+0000 7f9515ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.113+0000 7f9515ffb640 10 monclient(hunting): dump: 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:25.114 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:25.115 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:25.115 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.113+0000 7f9515ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:25.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.115+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446792240 0 0) 0x7f952406f0a0 con 0x7f95240af230 2024-03-25T19:51:25.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.115+0000 7f9515ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:25.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.115+0000 7f9515ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:25.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.115+0000 7f9515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:25.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.115+0000 7f9515ffb640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:51:25.116 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.115+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9508004eb0 con 0x7f95240af230 2024-03-25T19:51:25.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1967342731 0 0) 0x7f95240b00f0 con 0x7f952406fc40 2024-03-25T19:51:25.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 >> v1:172.21.15.110:6790/0 conn(0x7f95240af230 legacy=0x7f952406f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:25.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 >> v1:172.21.15.110:6789/0 conn(0x7f9524146680 legacy=0x7f952413f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:25.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:25.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 1 monclient: found mon.b 2024-03-25T19:51:25.118 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9524142e50 con 0x7f952406fc40 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f9515ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 5 monclient: authenticate success, global_id 4729 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 10 monclient: _renew_subs 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.178:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9524140730 con 0x7f952406fc40 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 10 monclient: _renew_subs 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.178:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9524140f90 con 0x7f952406fc40 2024-03-25T19:51:25.119 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.117+0000 7f952a39f640 1 librados: init done 2024-03-25T19:51:25.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.118+0000 7f952a39f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:25.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.118+0000 7f952a39f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:25.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.118+0000 7f952a39f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:25.120 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.118+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95240627b0 con 0x7f952406fc40 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.394+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f951c001210 con 0x7f952406fc40 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.394+0000 7f9515ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.395+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f951c004c30 con 0x7f952406fc40 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.395+0000 7f9515ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.395+0000 7f9515ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.395+0000 7f9515ffb640 10 monclient: dump: 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:25.396 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:25.397 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:25.397 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:25.397 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:25.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.395+0000 7f95177fe640 4 set_mon_vals no callback set 2024-03-25T19:51:25.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.396+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f951c01ac40 con 0x7f952406fc40 2024-03-25T19:51:25.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.396+0000 7f952939d640 10 monclient: get_auth_request con 0x7f950806ddb0 auth_method 0 2024-03-25T19:51:25.397 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.397+0000 7f9515ffb640 10 client.4729.objecter ms_handle_connect 0x7f950806ddb0 2024-03-25T19:51:25.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.400+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f951c0548d0 con 0x7f952406fc40 2024-03-25T19:51:25.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.400+0000 7f9515ffb640 10 client.4729.objecter ms_dispatch 0x7f9524000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:25.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.400+0000 7f9515ffb640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:25.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.400+0000 7f9515ffb640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:25.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.400+0000 7f9515ffb640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T19:51:25.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.400+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f951c050210 con 0x7f952406fc40 2024-03-25T19:51:25.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.400+0000 7f9515ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:25.401 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.400+0000 7f9515ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:25.658 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.657+0000 7f952a39f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:25.658 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.657+0000 7f952a39f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:25.658 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.657+0000 7f952a39f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.178:6789/0 2024-03-25T19:51:25.658 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.657+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 --> v1:172.21.15.178:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9524142fe0 con 0x7f952406fc40 2024-03-25T19:51:25.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.706+0000 7f9515ffb640 1 -- 172.21.15.110:0/1620674509 <== mon.1 v1:172.21.15.178:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f951c052310 con 0x7f952406fc40 2024-03-25T19:51:25.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.706+0000 7f9515ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:25.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.706+0000 7f9515ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:25.707 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:51:25.707 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:25.709 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 10 librados: watch_flush enter 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 10 librados: watch_flush exit 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 10 client.4729.objecter successfully canceled tick 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 >> v1:172.21.15.178:6800/1132739058 conn(0x7f950806ddb0 legacy=0x7f9508070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 10 monclient: shutdown 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 >> v1:172.21.15.178:6789/0 conn(0x7f952406fc40 legacy=0x7f9524140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 shutdown_connections 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.708+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 >> 172.21.15.110:0/1620674509 conn(0x7f95240b0320 msgr2=0x7f95241436d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.709+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 shutdown_connections 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.709+0000 7f952a39f640 1 -- 172.21.15.110:0/1620674509 wait complete. 2024-03-25T19:51:25.710 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:25.709+0000 7f952a39f640 1 librados: shutdown 2024-03-25T19:51:25.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:32.527 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:32.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.641+0000 7f59ac206640 10 monclient: get_monmap_and_config 2024-03-25T19:51:32.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.641+0000 7f59ac206640 10 monclient: build_initial_monmap 2024-03-25T19:51:32.641 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.641+0000 7f59ac206640 10 monclient: monmap: 2024-03-25T19:51:32.641 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:32.641 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:32.641 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:51:32.642622+0000 2024-03-25T19:51:32.641 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:51:32.642622+0000 2024-03-25T19:51:32.642 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:32.642 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:32.642 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:32.642 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:51:32.642 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:51:32.642 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:32.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.642+0000 7f59ac206640 1 Processor -- start 2024-03-25T19:51:32.642 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.642+0000 7f59ac206640 1 -- start start 2024-03-25T19:51:32.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.642+0000 7f59ac206640 10 monclient: init 2024-03-25T19:51:32.643 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 10 monclient(hunting): picked mon.noname-c con 0x7f59a40b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 10 monclient(hunting): picked mon.noname-a con 0x7f59a40b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 10 monclient(hunting): picked mon.noname-b con 0x7f59a414be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40af940 con 0x7f59a40b1730 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40b0a70 con 0x7f59a40b4d60 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40b0d50 con 0x7f59a414be90 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:32.644 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.643+0000 7f59ac206640 10 monclient(hunting): authenticate will time out at 4863.784758s 2024-03-25T19:51:32.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.650+0000 7f59aa7a3640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f59a414be90 0x7f59a414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:50790/0 (socket says 172.21.15.110:50790) 2024-03-25T19:51:32.650 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.650+0000 7f59aa7a3640 1 -- 172.21.15.110:0/1708408022 learned_addr learned my addr 172.21.15.110:0/1708408022 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:32.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.660+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811360101 0 0) 0x7f59a40af940 con 0x7f59a40b1730 2024-03-25T19:51:32.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.660+0000 7f59a8fa0640 10 monclient(hunting): _negotiate 2024-03-25T19:51:32.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.660+0000 7f59a8fa0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:32.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.660+0000 7f59a8fa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:32.659 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.660+0000 7f59a8fa0640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:51:32.660 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.660+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5990003520 con 0x7f59a40b1730 2024-03-25T19:51:32.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.661+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989772973 0 0) 0x7f59a40b0a70 con 0x7f59a40b4d60 2024-03-25T19:51:32.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.661+0000 7f59a8fa0640 10 monclient(hunting): _negotiate 2024-03-25T19:51:32.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.661+0000 7f59a8fa0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:32.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.661+0000 7f59a8fa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:32.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.661+0000 7f59a8fa0640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:51:32.662 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.661+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a40af940 con 0x7f59a40b4d60 2024-03-25T19:51:32.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431105408 0 0) 0x7f59a40b0d50 con 0x7f59a414be90 2024-03-25T19:51:32.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient(hunting): _negotiate 2024-03-25T19:51:32.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:32.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:32.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient(hunting): my global_id is 4733 2024-03-25T19:51:32.670 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a40b0a70 con 0x7f59a414be90 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3540940476 0 0) 0x7f5990003520 con 0x7f59a40b1730 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 >> v1:172.21.15.110:6790/0 conn(0x7f59a414be90 legacy=0x7f59a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 >> v1:172.21.15.178:6789/0 conn(0x7f59a40b4d60 legacy=0x7f59a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient: _finish_hunting 0 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 1 monclient: found mon.noname-a 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.110:6789/0 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a40b1030 con 0x7f59a40b1730 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient: _finish_auth 0 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient: _check_auth_tickets 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5994001690 con 0x7f59a40b1730 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59ac206640 5 monclient: authenticate success, global_id 4722 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59ac206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:32.671 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59a8fa0640 10 monclient: dump: 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:32.672 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.669+0000 7f59ac206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.679+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 <== mon.0 v1:172.21.15.110:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59940017f0 con 0x7f59a40b1730 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.679+0000 7f59a8fa0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.679+0000 7f59a8fa0640 1 -- 172.21.15.110:0/1708408022 <== mon.0 v1:172.21.15.110:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f5994001c00 con 0x7f59a40b1730 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.679+0000 7f59a8fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.679+0000 7f59a8fa0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.679+0000 7f59a8fa0640 10 monclient: dump: 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:32.679 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.679+0000 7f59ac206640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:32.680 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.679+0000 7f59ac206640 4 set_mon_vals no callback set 2024-03-25T19:51:32.681 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.680+0000 7f59ac206640 10 monclient: shutdown 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.680+0000 7f59ac206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.680+0000 7f59ac206640 1 -- 172.21.15.110:0/1708408022 >> v1:172.21.15.110:6789/0 conn(0x7f59a40b1730 legacy=0x7f59a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.680+0000 7f59ac206640 1 -- 172.21.15.110:0/1708408022 shutdown_connections 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.680+0000 7f59ac206640 1 -- 172.21.15.110:0/1708408022 >> 172.21.15.110:0/1708408022 conn(0x7f59a40617a0 msgr2=0x7f59a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.680+0000 7f59ac206640 1 -- 172.21.15.110:0/1708408022 shutdown_connections 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.680+0000 7f59ac206640 1 -- 172.21.15.110:0/1708408022 wait complete. 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.681+0000 7f59ac206640 10 monclient: build_initial_monmap 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.681+0000 7f59ac206640 10 monclient: monmap: 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:51:32.682 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.681+0000 7f59ac206640 1 librados: starting msgr at 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.681+0000 7f59ac206640 1 librados: starting objecter 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.682+0000 7f59ac206640 1 Processor -- start 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.682+0000 7f59ac206640 1 -- start start 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.682+0000 7f59ac206640 1 librados: setting wanted keys 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.682+0000 7f59ac206640 1 librados: calling monclient init 2024-03-25T19:51:32.683 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.682+0000 7f59ac206640 10 monclient: init 2024-03-25T19:51:32.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.682+0000 7f59ac206640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:32.684 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:32.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 10 monclient(hunting): picked mon.noname-a con 0x7f59a40b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:32.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 10 monclient(hunting): picked mon.noname-b con 0x7f59a40b4d60 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:32.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 10 monclient(hunting): picked mon.noname-c con 0x7f59a414be90 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:32.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a4144d40 con 0x7f59a40b1730 2024-03-25T19:51:32.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a41dbac0 con 0x7f59a40b4d60 2024-03-25T19:51:32.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a4135d30 con 0x7f59a414be90 2024-03-25T19:51:32.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:32.685 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.683+0000 7f59ac206640 10 monclient(hunting): authenticate will time out at 4863.824146s 2024-03-25T19:51:32.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.693+0000 7f59aa7a3640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f59a414be90 0x7f59a41e1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:50822/0 (socket says 172.21.15.110:50822) 2024-03-25T19:51:32.693 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.693+0000 7f59aa7a3640 1 -- 172.21.15.110:0/2465921319 learned_addr learned my addr 172.21.15.110:0/2465921319 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:32.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.697+0000 7f599a7fc640 10 client.?.objecter ms_handle_connect 0x7f59a414be90 2024-03-25T19:51:32.697 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.697+0000 7f599a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:32.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.700+0000 7f599a7fc640 10 client.?.objecter ms_handle_connect 0x7f59a40b1730 2024-03-25T19:51:32.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.700+0000 7f599a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:32.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.701+0000 7f599a7fc640 10 client.?.objecter ms_handle_connect 0x7f59a40b4d60 2024-03-25T19:51:32.701 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.701+0000 7f599a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:32.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.707+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767375893 0 0) 0x7f59a4135d30 con 0x7f59a414be90 2024-03-25T19:51:32.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.707+0000 7f599a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:32.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.707+0000 7f599a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:32.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.707+0000 7f599a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:32.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.707+0000 7f599a7fc640 10 monclient(hunting): my global_id is 4736 2024-03-25T19:51:32.707 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.707+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5988004eb0 con 0x7f59a414be90 2024-03-25T19:51:32.713 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.714+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935312646 0 0) 0x7f59a4144d40 con 0x7f59a40b1730 2024-03-25T19:51:32.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.714+0000 7f599a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:32.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.714+0000 7f599a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:32.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.714+0000 7f599a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:32.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.714+0000 7f599a7fc640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:51:32.714 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.714+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a4135d30 con 0x7f59a40b1730 2024-03-25T19:51:32.717 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.718+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766317211 0 0) 0x7f59a41dbac0 con 0x7f59a40b4d60 2024-03-25T19:51:32.718 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.718+0000 7f599a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:32.718 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.718+0000 7f599a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:32.718 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.718+0000 7f599a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:32.718 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.718+0000 7f599a7fc640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:51:32.718 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.718+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a4144d40 con 0x7f59a40b4d60 2024-03-25T19:51:32.719 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.719+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1626454839 0 0) 0x7f5988004eb0 con 0x7f59a414be90 2024-03-25T19:51:32.719 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.719+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59a41dbac0 con 0x7f59a414be90 2024-03-25T19:51:32.719 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.719+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f599c001bd0 con 0x7f59a414be90 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.719+0000 7f599a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.719+0000 7f599a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.719+0000 7f599a7fc640 10 monclient(hunting): dump: 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:32.720 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.720+0000 7f599a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:32.733 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3282172195 0 0) 0x7f59a41dbac0 con 0x7f59a414be90 2024-03-25T19:51:32.733 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 >> v1:172.21.15.178:6789/0 conn(0x7f59a40b4d60 legacy=0x7f59a4144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:32.733 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 >> v1:172.21.15.110:6789/0 conn(0x7f59a40b1730 legacy=0x7f59a41db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:32.733 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:32.733 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 1 monclient: found mon.c 2024-03-25T19:51:32.733 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a41382f0 con 0x7f59a414be90 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f599a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 5 monclient: authenticate success, global_id 4736 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 10 monclient: _renew_subs 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a4145670 con 0x7f59a414be90 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 10 client.4736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 10 monclient: _renew_subs 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a41447c0 con 0x7f59a414be90 2024-03-25T19:51:32.734 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.733+0000 7f59ac206640 1 librados: init done 2024-03-25T19:51:32.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.734+0000 7f59ac206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:32.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.734+0000 7f59ac206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:32.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.734+0000 7f59ac206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:32.735 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:32.734+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a40627b0 con 0x7f59a414be90 2024-03-25T19:51:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.031+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f599c002080 con 0x7f59a414be90 2024-03-25T19:51:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.031+0000 7f599a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.031+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f599c004d40 con 0x7f59a414be90 2024-03-25T19:51:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.031+0000 7f599a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.031+0000 7f599a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:33.031 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.031+0000 7f599a7fc640 10 monclient: dump: 2024-03-25T19:51:33.031 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:33.032 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.031+0000 7f599bfff640 4 set_mon_vals no callback set 2024-03-25T19:51:33.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.032+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f599c004f60 con 0x7f59a414be90 2024-03-25T19:51:33.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.033+0000 7f59a9fa2640 10 monclient: get_auth_request con 0x7f598806dd80 auth_method 0 2024-03-25T19:51:33.033 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.033+0000 7f599a7fc640 10 client.4736.objecter ms_handle_connect 0x7f598806dd80 2024-03-25T19:51:33.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.038+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f599c054a40 con 0x7f59a414be90 2024-03-25T19:51:33.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.038+0000 7f599a7fc640 10 client.4736.objecter ms_dispatch 0x7f59a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:33.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.038+0000 7f599a7fc640 3 client.4736.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:33.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.038+0000 7f599a7fc640 3 client.4736.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:33.038 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.038+0000 7f599a7fc640 20 client.4736.objecter dump_active .. 0 homeless 2024-03-25T19:51:33.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.038+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f599c050380 con 0x7f59a414be90 2024-03-25T19:51:33.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.038+0000 7f599a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:33.039 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.038+0000 7f599a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:33.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.288+0000 7f59ac206640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:33.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.288+0000 7f59ac206640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:33.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.288+0000 7f59ac206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:33.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.288+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59a4069f40 con 0x7f59a414be90 2024-03-25T19:51:33.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.323+0000 7f599a7fc640 1 -- 172.21.15.110:0/2465921319 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f599c052480 con 0x7f59a414be90 2024-03-25T19:51:33.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.323+0000 7f599a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:33.323 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.323+0000 7f599a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:33.323 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:51:33.324 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 10 librados: watch_flush enter 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 10 librados: watch_flush exit 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 20 client.4736.objecter shutdown clearing up homeless session... 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 10 client.4736.objecter successfully canceled tick 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 >> v1:172.21.15.178:6800/1132739058 conn(0x7f598806dd80 legacy=0x7f5988070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 10 monclient: shutdown 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 >> v1:172.21.15.110:6790/0 conn(0x7f59a414be90 legacy=0x7f59a41e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 shutdown_connections 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 >> 172.21.15.110:0/2465921319 conn(0x7f59a40617a0 msgr2=0x7f59a40b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 shutdown_connections 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.326+0000 7f59ac206640 1 -- 172.21.15.110:0/2465921319 wait complete. 2024-03-25T19:51:33.327 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:33.327+0000 7f59ac206640 1 librados: shutdown 2024-03-25T19:51:33.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:40.144 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:40.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.261+0000 7f6194f58640 10 monclient: get_monmap_and_config 2024-03-25T19:51:40.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.261+0000 7f6194f58640 10 monclient: build_initial_monmap 2024-03-25T19:51:40.260 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.261+0000 7f6194f58640 10 monclient: monmap: 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:51:40.261976+0000 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:51:40.261976+0000 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:51:40.261 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.261+0000 7f6194f58640 1 Processor -- start 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.261+0000 7f6194f58640 1 -- start start 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.261+0000 7f6194f58640 10 monclient: init 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 10 monclient(hunting): picked mon.noname-b con 0x7f61900b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 10 monclient(hunting): picked mon.noname-a con 0x7f61900b1730 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 10 monclient(hunting): picked mon.noname-c con 0x7f619014be90 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61900af940 con 0x7f61900b1730 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61900b0a70 con 0x7f619014be90 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61900b0d50 con 0x7f61900b4d60 2024-03-25T19:51:40.263 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:40.264 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.262+0000 7f6194f58640 10 monclient(hunting): authenticate will time out at 4871.403748s 2024-03-25T19:51:40.275 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.275+0000 7f618ed9d640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f619014be90 0x7f619014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:56266/0 (socket says 172.21.15.110:56266) 2024-03-25T19:51:40.275 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.275+0000 7f618ed9d640 1 -- 172.21.15.110:0/303896312 learned_addr learned my addr 172.21.15.110:0/303896312 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:40.301 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.302+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175680037 0 0) 0x7f61900b0a70 con 0x7f619014be90 2024-03-25T19:51:40.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.302+0000 7f618d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:40.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.302+0000 7f618d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:40.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.302+0000 7f618d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:40.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.302+0000 7f618d59a640 10 monclient(hunting): my global_id is 4738 2024-03-25T19:51:40.302 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.302+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6178003520 con 0x7f619014be90 2024-03-25T19:51:40.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.307+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779710894 0 0) 0x7f61900b0d50 con 0x7f61900b4d60 2024-03-25T19:51:40.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.307+0000 7f618d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:40.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.307+0000 7f618d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:40.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.307+0000 7f618d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:40.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.307+0000 7f618d59a640 10 monclient(hunting): my global_id is 4739 2024-03-25T19:51:40.307 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.307+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61900b0a70 con 0x7f61900b4d60 2024-03-25T19:51:40.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.309+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113579568 0 0) 0x7f61900af940 con 0x7f61900b1730 2024-03-25T19:51:40.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.309+0000 7f618d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:40.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.309+0000 7f618d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:40.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.309+0000 7f618d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:40.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.309+0000 7f618d59a640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:51:40.309 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.309+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61900b0d50 con 0x7f61900b1730 2024-03-25T19:51:40.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2831936851 0 0) 0x7f6178003520 con 0x7f619014be90 2024-03-25T19:51:40.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 >> v1:172.21.15.110:6790/0 conn(0x7f61900b4d60 legacy=0x7f619013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 >> v1:172.21.15.110:6789/0 conn(0x7f61900b1730 legacy=0x7f61900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:51:40.317 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 1 monclient: found mon.noname-c 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.178:6789/0 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61900b1030 con 0x7f619014be90 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 10 monclient: _finish_auth 0 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6188001a10 con 0x7f619014be90 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f6194f58640 5 monclient: authenticate success, global_id 4738 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f6194f58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f618d59a640 10 monclient: dump: 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:40.318 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:40.319 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.317+0000 7f6194f58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:40.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.334+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 <== mon.1 v1:172.21.15.178:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6188001b70 con 0x7f619014be90 2024-03-25T19:51:40.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.334+0000 7f618d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:40.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.334+0000 7f618d59a640 1 -- 172.21.15.110:0/303896312 <== mon.1 v1:172.21.15.178:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6188001f80 con 0x7f619014be90 2024-03-25T19:51:40.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.334+0000 7f6194f58640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:40.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.334+0000 7f6194f58640 4 set_mon_vals no callback set 2024-03-25T19:51:40.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.334+0000 7f618d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:40.334 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.334+0000 7f618d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.334+0000 7f618d59a640 10 monclient: dump: 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:40.335 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:40.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.335+0000 7f6194f58640 10 monclient: shutdown 2024-03-25T19:51:40.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.335+0000 7f6194f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:40.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.335+0000 7f6194f58640 1 -- 172.21.15.110:0/303896312 >> v1:172.21.15.178:6789/0 conn(0x7f619014be90 legacy=0x7f619014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.335+0000 7f6194f58640 1 -- 172.21.15.110:0/303896312 shutdown_connections 2024-03-25T19:51:40.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.335+0000 7f6194f58640 1 -- 172.21.15.110:0/303896312 >> 172.21.15.110:0/303896312 conn(0x7f61900617a0 msgr2=0x7f6190061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:40.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.336+0000 7f6194f58640 1 -- 172.21.15.110:0/303896312 shutdown_connections 2024-03-25T19:51:40.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.336+0000 7f6194f58640 1 -- 172.21.15.110:0/303896312 wait complete. 2024-03-25T19:51:40.336 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.336+0000 7f6194f58640 10 monclient: build_initial_monmap 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.336+0000 7f6194f58640 10 monclient: monmap: 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:51:40.337 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 1 librados: starting msgr at 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 1 librados: starting objecter 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 1 Processor -- start 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 1 -- start start 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 1 librados: setting wanted keys 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 1 librados: calling monclient init 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 10 monclient: init 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.337+0000 7f6194f58640 10 monclient(hunting): picked mon.noname-b con 0x7f61900b1730 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.338+0000 7f6194f58640 10 monclient(hunting): picked mon.noname-c con 0x7f61900b4d60 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.338+0000 7f6194f58640 10 monclient(hunting): picked mon.noname-a con 0x7f619014be90 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:40.339 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.338+0000 7f6194f58640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61901391b0 con 0x7f619014be90 2024-03-25T19:51:40.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.338+0000 7f6194f58640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6190132750 con 0x7f61900b1730 2024-03-25T19:51:40.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.338+0000 7f6194f58640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6190138e40 con 0x7f61900b4d60 2024-03-25T19:51:40.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.338+0000 7f6194f58640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:40.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.338+0000 7f6194f58640 10 monclient(hunting): authenticate will time out at 4871.479025s 2024-03-25T19:51:40.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.340+0000 7f618dd9b640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f61900b4d60 0x7f61901dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:55808/0 (socket says 172.21.15.110:55808) 2024-03-25T19:51:40.340 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.340+0000 7f618dd9b640 1 -- 172.21.15.110:0/2459395620 learned_addr learned my addr 172.21.15.110:0/2459395620 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:40.341 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.341+0000 7f617effd640 10 client.?.objecter ms_handle_connect 0x7f61900b4d60 2024-03-25T19:51:40.341 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.341+0000 7f617effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:40.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.352+0000 7f617effd640 10 client.?.objecter ms_handle_connect 0x7f619014be90 2024-03-25T19:51:40.352 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.352+0000 7f617effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:40.353 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.353+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725637192 0 0) 0x7f6190138e40 con 0x7f61900b4d60 2024-03-25T19:51:40.353 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.353+0000 7f617effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:40.353 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.353+0000 7f617effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:40.353 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.353+0000 7f617effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:40.353 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.353+0000 7f617effd640 10 monclient(hunting): my global_id is 4742 2024-03-25T19:51:40.354 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.353+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6170004eb0 con 0x7f61900b4d60 2024-03-25T19:51:40.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.359+0000 7f617effd640 10 client.?.objecter ms_handle_connect 0x7f61900b1730 2024-03-25T19:51:40.359 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.359+0000 7f617effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:40.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.363+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590524154 0 0) 0x7f61901391b0 con 0x7f619014be90 2024-03-25T19:51:40.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.363+0000 7f617effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:40.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.363+0000 7f617effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:40.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.363+0000 7f617effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:40.363 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.363+0000 7f617effd640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:51:40.364 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.363+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6190138e40 con 0x7f619014be90 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.370+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4179005754 0 0) 0x7f6170004eb0 con 0x7f61900b4d60 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.370+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61901391b0 con 0x7f61900b4d60 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.370+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6184001a10 con 0x7f61900b4d60 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.370+0000 7f617effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.370+0000 7f617effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.370+0000 7f617effd640 10 monclient(hunting): dump: 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:40.370 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:40.371 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:40.371 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:40.371 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:40.371 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:40.371 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.370+0000 7f617effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.378+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534652975 0 0) 0x7f6190138e40 con 0x7f619014be90 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.378+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6170004eb0 con 0x7f619014be90 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.378+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6188001e80 con 0x7f619014be90 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.378+0000 7f617effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.378+0000 7f617effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.378+0000 7f617effd640 10 monclient(hunting): dump: 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.378 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:40.379 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:40.379 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:40.379 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:40.379 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:40.379 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:40.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4284364701 0 0) 0x7f61901391b0 con 0x7f61900b4d60 2024-03-25T19:51:40.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 >> v1:172.21.15.178:6789/0 conn(0x7f61900b1730 legacy=0x7f6190132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 >> v1:172.21.15.110:6789/0 conn(0x7f619014be90 legacy=0x7f61901db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:40.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 1 monclient: found mon.c 2024-03-25T19:51:40.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:40.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:40.382 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6190137c00 con 0x7f61900b4d60 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 10 monclient: _finish_auth 0 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f617effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 5 monclient: authenticate success, global_id 4742 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 10 monclient: _renew_subs 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6190138050 con 0x7f61900b4d60 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 10 client.4742.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 10 monclient: _renew_subs 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6190145080 con 0x7f61900b4d60 2024-03-25T19:51:40.383 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.382+0000 7f6194f58640 1 librados: init done 2024-03-25T19:51:40.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.383+0000 7f6194f58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:40.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.383+0000 7f6194f58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:40.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.383+0000 7f6194f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:40.384 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.383+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61901401f0 con 0x7f61900b4d60 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.698+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6184001c70 con 0x7f61900b4d60 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.698+0000 7f617effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.698+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f6184003a60 con 0x7f61900b4d60 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.698+0000 7f617effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.698+0000 7f617effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.698+0000 7f617effd640 10 monclient: dump: 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:40.698 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.699 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:40.699 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:40.699 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:40.699 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:40.699 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:40.699 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:40.699 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:40.699 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.698+0000 7f618cd99640 4 set_mon_vals no callback set 2024-03-25T19:51:40.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.699+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f6184003c80 con 0x7f61900b4d60 2024-03-25T19:51:40.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.700+0000 7f618e59c640 10 monclient: get_auth_request con 0x7f617006de90 auth_method 0 2024-03-25T19:51:40.700 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.700+0000 7f617effd640 10 client.4742.objecter ms_handle_connect 0x7f617006de90 2024-03-25T19:51:40.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.706+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f6184054880 con 0x7f61900b4d60 2024-03-25T19:51:40.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.706+0000 7f617effd640 10 client.4742.objecter ms_dispatch 0x7f6190000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:40.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.706+0000 7f617effd640 3 client.4742.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:40.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.706+0000 7f617effd640 3 client.4742.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:40.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.706+0000 7f617effd640 20 client.4742.objecter dump_active .. 0 homeless 2024-03-25T19:51:40.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.706+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6184050390 con 0x7f61900b4d60 2024-03-25T19:51:40.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.706+0000 7f617effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:40.706 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.706+0000 7f617effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:40.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.930+0000 7f6194f58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:40.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.930+0000 7f6194f58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:40.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.930+0000 7f6194f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.110:6790/0 2024-03-25T19:51:40.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.930+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 --> v1:172.21.15.110:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6190138fd0 con 0x7f61900b4d60 2024-03-25T19:51:40.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.956+0000 7f617effd640 1 -- 172.21.15.110:0/2459395620 <== mon.2 v1:172.21.15.110:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f61840522c0 con 0x7f61900b4d60 2024-03-25T19:51:40.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.956+0000 7f617effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:40.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.956+0000 7f617effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:40.956 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:51:40.956 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 10 librados: watch_flush enter 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 10 librados: watch_flush exit 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 20 client.4742.objecter shutdown clearing up homeless session... 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 10 client.4742.objecter successfully canceled tick 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 >> v1:172.21.15.178:6800/1132739058 conn(0x7f617006de90 legacy=0x7f6170070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 10 monclient: shutdown 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 >> v1:172.21.15.110:6790/0 conn(0x7f61900b4d60 legacy=0x7f61901dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 shutdown_connections 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 >> 172.21.15.110:0/2459395620 conn(0x7f61900617a0 msgr2=0x7f61900b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 shutdown_connections 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 1 -- 172.21.15.110:0/2459395620 wait complete. 2024-03-25T19:51:40.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:40.958+0000 7f6194f58640 1 librados: shutdown 2024-03-25T19:51:40.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:44.355 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:51.639 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.416 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:51.700 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.448 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:51.732 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.453 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:51.708 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.456 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:51.711 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.473 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:51.757 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.484 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:51.768 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.498 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:11:51.752 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.512 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:51.796 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:47.777 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:47.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.892+0000 7f9bebd39640 10 monclient: get_monmap_and_config 2024-03-25T19:51:47.892 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.892+0000 7f9bebd39640 10 monclient: build_initial_monmap 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.892+0000 7f9bebd39640 10 monclient: monmap: 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:51:47.893888+0000 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:51:47.893888+0000 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-c 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-b 2024-03-25T19:51:47.893 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:47.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.893+0000 7f9bebd39640 1 Processor -- start 2024-03-25T19:51:47.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.893+0000 7f9bebd39640 1 -- start start 2024-03-25T19:51:47.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.893+0000 7f9bebd39640 10 monclient: init 2024-03-25T19:51:47.894 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 10 monclient(hunting): picked mon.noname-b con 0x7f9be40a98f0 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 10 monclient(hunting): picked mon.noname-a con 0x7f9be413f710 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 10 monclient(hunting): picked mon.noname-c con 0x7f9be41430c0 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be4142520 con 0x7f9be413f710 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be40b28a0 con 0x7f9be41430c0 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be40b2a30 con 0x7f9be40a98f0 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:47.895 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.894+0000 7f9bebd39640 10 monclient(hunting): authenticate will time out at 4879.035402s 2024-03-25T19:51:47.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.900+0000 7f9be9ad5640 1 --1- >> v1:172.21.15.110:6790/0 conn(0x7f9be40a98f0 0x7f9be40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6790/0 says I am v1:172.21.15.110:35496/0 (socket says 172.21.15.110:35496) 2024-03-25T19:51:47.900 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.900+0000 7f9be9ad5640 1 -- 172.21.15.110:0/3304886042 learned_addr learned my addr 172.21.15.110:0/3304886042 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:47.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.914+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 <== mon.2 v1:172.21.15.110:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862821366 0 0) 0x7f9be40b2a30 con 0x7f9be40a98f0 2024-03-25T19:51:47.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.914+0000 7f9be8ad3640 10 monclient(hunting): _negotiate 2024-03-25T19:51:47.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.914+0000 7f9be8ad3640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:47.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.914+0000 7f9be8ad3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:47.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.914+0000 7f9be8ad3640 10 monclient(hunting): my global_id is 4745 2024-03-25T19:51:47.914 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.914+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 --> v1:172.21.15.110:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd0003520 con 0x7f9be40a98f0 2024-03-25T19:51:47.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.925+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647986597 0 0) 0x7f9be40b28a0 con 0x7f9be41430c0 2024-03-25T19:51:47.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.925+0000 7f9be8ad3640 10 monclient(hunting): _negotiate 2024-03-25T19:51:47.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.925+0000 7f9be8ad3640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:47.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.925+0000 7f9be8ad3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:47.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.925+0000 7f9be8ad3640 10 monclient(hunting): my global_id is 4744 2024-03-25T19:51:47.925 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.925+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be40b2a30 con 0x7f9be41430c0 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 <== mon.2 v1:172.21.15.110:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2307308685 0 0) 0x7f9bd0003520 con 0x7f9be40a98f0 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 >> v1:172.21.15.178:6789/0 conn(0x7f9be41430c0 legacy=0x7f9be414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 >> v1:172.21.15.110:6789/0 conn(0x7f9be413f710 legacy=0x7f9be41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 10 monclient: _finish_hunting 0 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 1 monclient: found mon.noname-b 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.110:6790/0 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 --> v1:172.21.15.110:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be40b44c0 con 0x7f9be40a98f0 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 10 monclient: _finish_auth 0 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 10 monclient: _check_auth_tickets 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:47.930 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 <== mon.2 v1:172.21.15.110:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9be0001690 con 0x7f9be40a98f0 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9be8ad3640 10 monclient: dump: 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9bebd39640 5 monclient: authenticate success, global_id 4745 2024-03-25T19:51:47.931 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.930+0000 7f9bebd39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:47.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.938+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 <== mon.2 v1:172.21.15.110:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9be00017f0 con 0x7f9be40a98f0 2024-03-25T19:51:47.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.938+0000 7f9be8ad3640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:47.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.938+0000 7f9be8ad3640 1 -- 172.21.15.110:0/3304886042 <== mon.2 v1:172.21.15.110:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9be0001c00 con 0x7f9be40a98f0 2024-03-25T19:51:47.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.938+0000 7f9bebd39640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:47.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.938+0000 7f9bebd39640 4 set_mon_vals no callback set 2024-03-25T19:51:47.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.938+0000 7f9be8ad3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:47.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.938+0000 7f9be8ad3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:47.938 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.938+0000 7f9be8ad3640 10 monclient: dump: 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:47.939 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:47.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.939+0000 7f9bebd39640 10 monclient: shutdown 2024-03-25T19:51:47.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.939+0000 7f9bebd39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:47.940 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.939+0000 7f9bebd39640 1 -- 172.21.15.110:0/3304886042 >> v1:172.21.15.110:6790/0 conn(0x7f9be40a98f0 legacy=0x7f9be40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.939+0000 7f9bebd39640 1 -- 172.21.15.110:0/3304886042 shutdown_connections 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.939+0000 7f9bebd39640 1 -- 172.21.15.110:0/3304886042 >> 172.21.15.110:0/3304886042 conn(0x7f9be40b7ea0 msgr2=0x7f9be40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.939+0000 7f9bebd39640 1 -- 172.21.15.110:0/3304886042 shutdown_connections 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.939+0000 7f9bebd39640 1 -- 172.21.15.110:0/3304886042 wait complete. 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.940+0000 7f9bebd39640 10 monclient: build_initial_monmap 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.940+0000 7f9bebd39640 10 monclient: monmap: 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a 2024-03-25T19:51:47.941 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.noname-b 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.noname-c 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.940+0000 7f9bebd39640 1 librados: starting msgr at 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.940+0000 7f9bebd39640 1 librados: starting objecter 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.940+0000 7f9bebd39640 1 Processor -- start 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 1 -- start start 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 1 librados: setting wanted keys 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 1 librados: calling monclient init 2024-03-25T19:51:47.942 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 10 monclient: init 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 10 monclient(hunting): picked mon.noname-b con 0x7f9be40a98f0 addr v1:172.21.15.178:6789/0 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 10 monclient(hunting): picked mon.noname-c con 0x7f9be413f710 addr v1:172.21.15.110:6790/0 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 10 monclient(hunting): picked mon.noname-a con 0x7f9be41430c0 addr v1:172.21.15.110:6789/0 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be40ab9e0 con 0x7f9be41430c0 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be4138c20 con 0x7f9be40a98f0 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 1 -- --> v1:172.21.15.110:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be4138f00 con 0x7f9be413f710 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:47.943 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.941+0000 7f9bebd39640 10 monclient(hunting): authenticate will time out at 4879.082811s 2024-03-25T19:51:47.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.954+0000 7f9be9ad5640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f9be40a98f0 0x7f9be4137fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.110:39942/0 (socket says 172.21.15.110:39942) 2024-03-25T19:51:47.954 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.954+0000 7f9be9ad5640 1 -- 172.21.15.110:0/1108132164 learned_addr learned my addr 172.21.15.110:0/1108132164 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-03-25T19:51:47.955 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.956+0000 7f9bda7fc640 10 client.?.objecter ms_handle_connect 0x7f9be40a98f0 2024-03-25T19:51:47.956 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.956+0000 7f9bda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:47.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.960+0000 7f9bda7fc640 10 client.?.objecter ms_handle_connect 0x7f9be41430c0 2024-03-25T19:51:47.959 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.960+0000 7f9bda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:47.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 client.?.objecter ms_handle_connect 0x7f9be413f710 2024-03-25T19:51:47.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:47.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695409066 0 0) 0x7f9be4138c20 con 0x7f9be40a98f0 2024-03-25T19:51:47.963 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 monclient(hunting): my global_id is 4747 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc8004eb0 con 0x7f9be40a98f0 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174743130 0 0) 0x7f9be40ab9e0 con 0x7f9be41430c0 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:51:47.964 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.963+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be4138c20 con 0x7f9be41430c0 2024-03-25T19:51:47.976 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.977+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3439937247 0 0) 0x7f9be4138c20 con 0x7f9be41430c0 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.977+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9be40ab9e0 con 0x7f9be41430c0 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.977+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9bdc001b90 con 0x7f9be41430c0 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.977+0000 7f9bda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.977+0000 7f9bda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.977+0000 7f9bda7fc640 10 monclient(hunting): dump: 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:47.977 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.977+0000 7f9bda7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:47.983 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 49742427 0 0) 0x7f9be40ab9e0 con 0x7f9be41430c0 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 >> v1:172.21.15.110:6790/0 conn(0x7f9be413f710 legacy=0x7f9be41d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 >> v1:172.21.15.178:6789/0 conn(0x7f9be40a98f0 legacy=0x7f9be4137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 1 monclient: found mon.a 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be414e7a0 con 0x7f9be41430c0 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bebd39640 5 monclient: authenticate success, global_id 4737 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bebd39640 10 monclient: _renew_subs 2024-03-25T19:51:47.984 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bebd39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.983+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 --> v1:172.21.15.110:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be41386d0 con 0x7f9be41430c0 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.984+0000 7f9bebd39640 10 client.4737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.984+0000 7f9bebd39640 10 monclient: _renew_subs 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.984+0000 7f9bebd39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.984+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 --> v1:172.21.15.110:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be41d8060 con 0x7f9be41430c0 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.984+0000 7f9bebd39640 1 librados: init done 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.985+0000 7f9bebd39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.985+0000 7f9bebd39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:47.985 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.985+0000 7f9bebd39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:47.986 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:47.985+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be40627b0 con 0x7f9be41430c0 2024-03-25T19:51:48.286 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.286+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bdc002040 con 0x7f9be41430c0 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.286+0000 7f9bda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.286+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 484414356 0 0) 0x7f9bdc003b30 con 0x7f9be41430c0 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.286+0000 7f9bda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.286+0000 7f9bda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.286+0000 7f9bda7fc640 10 monclient: dump: 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:fsid 042bf057-0adf-4f81-8758-4792528254d3 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:created 2024-03-25T19:39:23.663646+0000 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 3 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.a 2024-03-25T19:51:48.287 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.178:6789/0 mon.b 2024-03-25T19:51:48.288 INFO:teuthology.misc.health.smithi110.stderr:2: v1:172.21.15.110:6790/0 mon.c 2024-03-25T19:51:48.288 INFO:teuthology.misc.health.smithi110.stderr: 2024-03-25T19:51:48.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.286+0000 7f9bdbfff640 4 set_mon_vals no callback set 2024-03-25T19:51:48.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.287+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1681385574 0 0) 0x7f9bdc019b00 con 0x7f9be41430c0 2024-03-25T19:51:48.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.288+0000 7f9be9ad5640 10 monclient: get_auth_request con 0x7f9bc806dd40 auth_method 0 2024-03-25T19:51:48.288 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.288+0000 7f9bda7fc640 10 client.4737.objecter ms_handle_connect 0x7f9bc806dd40 2024-03-25T19:51:48.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.293+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 505851106 0 0) 0x7f9bdc053810 con 0x7f9be41430c0 2024-03-25T19:51:48.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.293+0000 7f9bda7fc640 10 client.4737.objecter ms_dispatch 0x7f9be4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:48.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.293+0000 7f9bda7fc640 3 client.4737.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:48.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.293+0000 7f9bda7fc640 3 client.4737.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:48.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.293+0000 7f9bda7fc640 20 client.4737.objecter dump_active .. 0 homeless 2024-03-25T19:51:48.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.293+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9bdc04f150 con 0x7f9be41430c0 2024-03-25T19:51:48.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.293+0000 7f9bda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:48.293 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.293+0000 7f9bda7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:48.541 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.542+0000 7f9bebd39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:48.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.542+0000 7f9bebd39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:48.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.542+0000 7f9bebd39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.110:6789/0 2024-03-25T19:51:48.542 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.542+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 --> v1:172.21.15.110:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9be414f240 con 0x7f9be41430c0 2024-03-25T19:51:48.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.564+0000 7f9bda7fc640 1 -- 172.21.15.110:0/1108132164 <== mon.0 v1:172.21.15.110:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3211275583) 0x7f9bdc051250 con 0x7f9be41430c0 2024-03-25T19:51:48.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.564+0000 7f9bda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:48.564 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.564+0000 7f9bda7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:48.564 INFO:teuthology.misc.health.smithi110.stdout: 2024-03-25T19:51:48.565 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":11,"fsid":"042bf057-0adf-4f81-8758-4792528254d3","created":"2024-03-25T19:40:19.870065+0000","modified":"2024-03-25T19:50:24.052669+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:24.052669+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"048b4cae-9f46-46e8-a412-3809fe6e4b7b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d209ed26-5603-4a13-a52f-7a40169d3c6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"39e9bb9d-3b9c-47bb-a410-56019ae94e67","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9ebeaa83-4e6a-4770-9656-4f0d65dd06fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f422fd1b-9010-47e4-8509-0ad530571a80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c08330a9-07e8-49ab-b0c7-fbade08e842d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"57e9f7f2-6fb8-4958-9cae-c8218eebd46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9e856141-e93f-4a86-bcbc-2e6e5778e31c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:48.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 10 librados: watch_flush enter 2024-03-25T19:51:48.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 10 librados: watch_flush exit 2024-03-25T19:51:48.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 20 client.4737.objecter shutdown clearing up homeless session... 2024-03-25T19:51:48.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 10 client.4737.objecter successfully canceled tick 2024-03-25T19:51:48.567 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 >> v1:172.21.15.178:6800/1132739058 conn(0x7f9bc806dd40 legacy=0x7f9bc8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:48.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 10 monclient: shutdown 2024-03-25T19:51:48.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:48.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 >> v1:172.21.15.110:6789/0 conn(0x7f9be41430c0 legacy=0x7f9be41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:48.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 shutdown_connections 2024-03-25T19:51:48.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 >> 172.21.15.110:0/1108132164 conn(0x7f9be40b7ea0 msgr2=0x7f9be4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:48.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 shutdown_connections 2024-03-25T19:51:48.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 1 -- 172.21.15.110:0/1108132164 wait complete. 2024-03-25T19:51:48.568 INFO:teuthology.misc.health.smithi110.stderr:2024-03-25T19:51:48.567+0000 7f9bebd39640 1 librados: shutdown 2024-03-25T19:51:48.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:48.581 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:51:48.583 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:51:48.583 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:51:48.583 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:51:50.685 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:11:24.837 35560== 2024-03-25T19:51:50.685 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:11:24.837 35560== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:51:50.724 INFO:tasks.ceph.osd.0.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:51:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:51:54.685 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:51:54.686 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:51:54.686 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:51:54.686 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:51:56.597 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:11:30.756 35559== 2024-03-25T19:51:56.597 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:11:30.756 35559== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:51:56.639 INFO:tasks.ceph.osd.1.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:00.788 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:52:00.788 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:52:00.788 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:52:00.789 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:02.614 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:11:36.762 35562== 2024-03-25T19:52:02.614 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:11:36.762 35562== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:02.652 INFO:tasks.ceph.osd.2.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:04.450 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:11.734 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.456 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:11.740 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.484 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:11.768 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.506 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:11.761 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.512 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:11.796 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.514 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:11.768 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.516 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:11.800 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.540 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:11.795 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.555 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:11.839 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.890 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:06.891 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:52:06.891 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:52:06.891 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:52:06.891 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:08.626 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:11:42.769 35564== 2024-03-25T19:52:08.626 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:11:42.769 35564== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:08.668 INFO:tasks.ceph.osd.3.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:12.993 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:52:12.994 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:52:12.994 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:52:12.994 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:14.926 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:11:49.046 35313== 2024-03-25T19:52:14.926 INFO:tasks.ceph.osd.4.smithi178.stderr:==00:00:11:49.046 35313== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:14.961 INFO:tasks.ceph.osd.4.smithi178.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:19.096 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:52:19.096 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:52:19.097 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:52:19.097 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:21.070 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:11:55.196 35305== 2024-03-25T19:52:21.070 INFO:tasks.ceph.osd.5.smithi178.stderr:==00:00:11:55.196 35305== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:21.109 INFO:tasks.ceph.osd.5.smithi178.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:24.581 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:31.865 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.585 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:31.869 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.608 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:31.892 35251== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.632 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:31.887 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.636 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:31.890 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.644 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:31.928 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.646 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:31.901 34704== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.649 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:31.933 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.659 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:31.943 35250== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:25.198 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:52:25.199 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:52:25.199 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:52:25.199 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:26.911 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:12:01.034 35311== 2024-03-25T19:52:26.911 INFO:tasks.ceph.osd.6.smithi178.stderr:==00:00:12:01.034 35311== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:26.957 INFO:tasks.ceph.osd.6.smithi178.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:31.301 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:52:31.301 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:52:31.301 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:52:31.302 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:32.738 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:12:06.845 35329== 2024-03-25T19:52:32.738 INFO:tasks.ceph.osd.7.smithi178.stderr:==00:00:12:06.845 35329== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:32.771 INFO:tasks.ceph.osd.7.smithi178.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:37.403 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:52:37.404 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:52:37.404 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:52:37.404 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:52:37.404 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:37.431 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:52:37.431 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:52:37.432 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:37.433 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:52:37.433+0000 7fb50b714640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34702) UID: 0 2024-03-25T19:52:37.433 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-25T19:52:37.433+0000 7fb50b714640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:52:37.460 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:52:37.460 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:52:37.461 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:52:37.461 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:37.487 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-03-25T19:52:37.476+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35216) UID: 0 2024-03-25T19:52:37.487 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-03-25T19:52:37.478+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:52:37.954 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:45.238 35251== 2024-03-25T19:52:37.954 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:45.238 35251== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:37.963 INFO:tasks.ceph.mon.a.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:43.563 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:52:43.564 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:52:43.564 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:52:43.564 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:43.580 INFO:tasks.ceph.mon.c.smithi110.stderr:2024-03-25T19:52:43.569+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35218) UID: 0 2024-03-25T19:52:43.580 INFO:tasks.ceph.mon.c.smithi110.stderr:2024-03-25T19:52:43.571+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:52:43.991 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:51.275 35250== 2024-03-25T19:52:43.992 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:51.275 35250== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:44.001 INFO:tasks.ceph.mon.c.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:49.666 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:52:49.667 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:52:49.667 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:52:49.667 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:49.682 INFO:tasks.ceph.mon.b.smithi178.stderr:2024-03-25T19:52:49.672+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34700) UID: 0 2024-03-25T19:52:49.683 INFO:tasks.ceph.mon.b.smithi178.stderr:2024-03-25T19:52:49.673+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:52:50.338 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:57.592 34704== 2024-03-25T19:52:50.338 INFO:tasks.ceph.mon.b.smithi178.stderr:==00:00:12:57.592 34704== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:50.347 INFO:tasks.ceph.mon.b.smithi178.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:55.768 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:52:55.769 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:52:55.769 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:52:55.769 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:52:55.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:52:55.882 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:52:56.018 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:52:56.018 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:52:56.116 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:52:56.116 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:52:56.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:52:56.194 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:52:56.265 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:52:56.265 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:52:56.360 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:52:56.360 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:52:56.450 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:52:56.450 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:52:56.507 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:52:56.507 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:52:56.573 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:52:56.574 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/data/mon.a.tgz 2024-03-25T19:52:56.574 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-03-25T19:52:56.590 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.rd5mFEIN8O 2024-03-25T19:52:56.590 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rd5mFEIN8O 2024-03-25T19:52:56.695 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.rd5mFEIN8O 2024-03-25T19:52:56.782 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.rd5mFEIN8O is 134KB 2024-03-25T19:52:56.834 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.rd5mFEIN8O 2024-03-25T19:52:56.849 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/data/mon.c.tgz 2024-03-25T19:52:56.849 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-03-25T19:52:56.903 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.K1m3wSGbn5 2024-03-25T19:52:56.903 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.K1m3wSGbn5 2024-03-25T19:52:56.991 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.K1m3wSGbn5 2024-03-25T19:52:57.075 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.K1m3wSGbn5 is 134KB 2024-03-25T19:52:57.125 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.K1m3wSGbn5 2024-03-25T19:52:57.139 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/data/mon.b.tgz 2024-03-25T19:52:57.140 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-03-25T19:52:57.157 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.dD9KVA6An6 2024-03-25T19:52:57.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.dD9KVA6An6 2024-03-25T19:52:57.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.dD9KVA6An6 2024-03-25T19:52:57.373 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.dD9KVA6An6 is 134KB 2024-03-25T19:52:57.424 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.dD9KVA6An6 2024-03-25T19:52:57.439 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:52:57.439 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:52:57.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:52:57.796 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:52:58.042 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:52:58.044 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/remote/smithi110/crash 2024-03-25T19:52:58.045 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:52:58.071 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/remote/smithi178/crash 2024-03-25T19:52:58.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:52:58.098 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:52:58.099 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:52:58.115 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:52:58.190 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:52:58.190 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34941.log 2024-03-25T19:52:58.190 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:52:58.190 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34941.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:52:58.190 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34941.log.gz 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34901.log 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35112.log 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34901.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35112.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34901.log.gz 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35065.log 2024-03-25T19:52:58.191 INFO:teuthology.orchestra.run.smithi178.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35112.log.gz 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5/var/log/ceph/ceph-osd.4.log: --verbose -- /var/log/ceph/ceph-client.admin.34861.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34981.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35065.log: /var/log/ceph/ceph-client.admin.34861.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35065.log.gz 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.34981.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34861.log.gz 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34981.log.gz 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35021.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.35021.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35021.log.gz 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35152.log 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.35152.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:52:58.192 INFO:teuthology.orchestra.run.smithi178.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35152.log.gz 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40029.log 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38862.log 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41082.log 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38862.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40029.log.gz 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39676.log 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38862.log.gz 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39169.log 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39598.log 2024-03-25T19:52:58.194 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40224.log 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39676.log.gz 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39169.log.gz 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40224.log: 86.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.39598.log.gz -5 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42294.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40224.log.gz 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:52:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40107.log 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39325.log 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40068.log 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40107.log.gz 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39559.log 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39325.log.gz 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35589.log 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40068.log.gz 2024-03-25T19:52:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41356.log 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39559.log.gz 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41122.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35589.log.gz 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41356.log.gz 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41902.log 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40887.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39520.log 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40536.log 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39520.log: gzip -5 --verbose 86.6% -- -- replaced with /var/log/ceph/ceph-client.admin.41902.log.gz /var/log/ceph/ceph-client.admin.39089.log 2024-03-25T19:52:58.197 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40731.log 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40536.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39520.log.gz 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40926.log 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40536.log.gz 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37582.log 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39089.log.gz 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41395.log 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.37582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42372.log 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40926.log.gz 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41395.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37582.log.gz 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41395.log.gz 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42099.log 2024-03-25T19:52:58.198 INFO:teuthology.orchestra.run.smithi110.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42372.log.gz 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42021.log 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42099.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41707.log 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42099.log.gz 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39208.log 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42021.log: /var/log/ceph/ceph-client.admin.41707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42060.log 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42021.log.gz 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40380.log 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41707.log.gz 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39208.log.gz 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41943.log 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42060.log.gz 2024-03-25T19:52:58.199 INFO:teuthology.orchestra.run.smithi110.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40380.log.gz 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40809.log 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39442.log 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40497.log 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41943.log.gz 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39990.log 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40809.log.gz 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35566.log 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39442.log.gz 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39990.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40497.log.gz 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35566.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39990.log.gz 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39364.log 2024-03-25T19:52:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39128.log 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.35566.log.gz 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41551.log 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39128.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39364.log.gz 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40653.log 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41551.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39128.log.gz/var/log/ceph/ceph-client.admin.40653.log: 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40302.log 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41551.log.gz 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41668.log 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40653.log.gz 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39832.log 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40302.log.gz 2024-03-25T19:52:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39832.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41668.log.gz 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42138.log 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39754.log 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39637.log 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39754.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39247.log 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42138.log.gz 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39754.log.gz 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39637.log.gz 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41317.log: gzip -5 --verbose -- 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39247.log.gz 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr: /var/log/ceph/ceph-client.admin.41863.log 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40185.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2024-03-25T19:52:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40419.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41863.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39481.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr: 86.2%/var/log/ceph/ceph-client.admin.40419.log: -- replaced with /var/log/ceph/ceph-client.admin.41863.log.gz 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40146.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39481.log: gzip -5 --verbose 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40419.log.gz 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.41512.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8%/var/log/ceph/ceph-client.admin.40146.log: -- replaced with /var/log/ceph/ceph-client.admin.39481.log.gz 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39403.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41512.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40146.log.gz 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41512.log.gz 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42333.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose/var/log/ceph/ceph-mgr.y.log: -- /var/log/ceph/ceph-client.admin.41785.log 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39403.log.gz 2024-03-25T19:52:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41004.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42333.log.gz 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39715.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40965.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41785.log.gz 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41004.log.gz 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40965.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39715.log.gz 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41161.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40965.log.gz 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41161.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42177.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41200.log: gzip 87.2% -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41161.log.gz 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41590.log 2024-03-25T19:52:58.204 INFO:teuthology.orchestra.run.smithi110.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gz 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41982.log: gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42177.log.gz 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41629.log 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41590.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41629.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41590.log.gz 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.41746.log 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.0.log: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41629.log.gz 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40458.log: -- /var/log/ceph/ceph.log 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41746.log.gz 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40341.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41824.log 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:52:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41824.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41824.log.gz 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35733.log 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40575.log 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35733.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5%/var/log/ceph/ceph-client.admin.40575.log: -- replaced with /var/log/ceph/ceph-client.admin.35733.log.gz 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40575.log.gz 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42255.log 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41278.log 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.42255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41239.log 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41278.log: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42255.log.gz 2024-03-25T19:52:58.206 INFO:teuthology.orchestra.run.smithi110.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41278.log.gz 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39793.log 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40614.log 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35258.log 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40614.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41239.log.gz 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39949.log 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35258.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40614.log.gz 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39793.log.gz 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29407.log 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39871.log 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35258.log.gz 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.tmp-client.admin.29407.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39949.log.gz 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29407.log.gz 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41473.log 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39871.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:52:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41473.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36599.log 2024-03-25T19:52:58.208 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39871.log.gz 2024-03-25T19:52:58.208 INFO:teuthology.orchestra.run.smithi110.stderr: 86.6%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.41473.log.gz 2024-03-25T19:52:58.208 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36599.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36599.log.gz 2024-03-25T19:52:58.229 INFO:teuthology.orchestra.run.smithi178.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:52:58.979 INFO:teuthology.orchestra.run.smithi110.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:52:58.998 INFO:teuthology.orchestra.run.smithi110.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:52:59.079 INFO:teuthology.orchestra.run.smithi178.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:52:59.109 INFO:teuthology.orchestra.run.smithi178.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:52:59.281 INFO:teuthology.orchestra.run.smithi110.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:52:59.310 INFO:teuthology.orchestra.run.smithi110.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:52:59.362 INFO:teuthology.orchestra.run.smithi178.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:52:59.430 INFO:teuthology.orchestra.run.smithi178.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:53:03.273 INFO:teuthology.orchestra.run.smithi110.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:53:03.274 INFO:teuthology.orchestra.run.smithi178.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:53:03.276 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-25T19:53:03.276 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m5.115s 2024-03-25T19:53:03.276 INFO:teuthology.orchestra.run.smithi178.stderr:user 0m9.109s 2024-03-25T19:53:03.276 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m0.343s 2024-03-25T19:53:03.415 INFO:teuthology.orchestra.run.smithi110.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:53:03.417 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-03-25T19:53:03.417 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m5.290s 2024-03-25T19:53:03.417 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m13.872s 2024-03-25T19:53:03.417 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m0.538s 2024-03-25T19:53:03.417 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:53:03.417 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/remote/smithi110/log 2024-03-25T19:53:03.418 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:53:05.897 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/remote/smithi178/log 2024-03-25T19:53:05.897 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:53:07.468 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:53:07.660 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=baaf4fb94f5d4fcf9ba9d180953680c7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:53:07.664 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:53:07.673 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:53:07.680 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:53:07.681 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:53:07.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:53:07.773 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:53:07.781 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:53:07.781 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:53:07.784 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:53:07.797 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:53:07.800 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:53:07.803 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:53:07.803 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-03-25T19:53:07.803 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 -554us[ -554us] +/- 79ms 2024-03-25T19:53:07.803 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -245us[ -254us] +/- 65ms 2024-03-25T19:53:07.803 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +710us[ +710us] +/- 68ms 2024-03-25T19:53:07.803 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:53:07.805 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:53:07.806 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-03-25T19:53:07.806 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 -548us[ -557us] +/- 79ms 2024-03-25T19:53:07.806 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -242us[ -251us] +/- 65ms 2024-03-25T19:53:07.806 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +715us[ +706us] +/- 68ms 2024-03-25T19:53:07.806 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:53:07.806 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:53:07.815 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:53:07.815 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:53:07.822 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:53:07.913 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:53:07.983 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:53:08.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:53:08.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:53:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:53:08.044 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:53:08.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:53:08.060 INFO:teuthology.task.internal:Duration was 4583.020409 seconds 2024-03-25T19:53:08.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:53:08.069 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:53:08.070 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:53:08.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:53:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:53:08.128 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:53:08.530 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:53:08.530 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-03-25T19:53:08.531 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:53:08.550 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-03-25T19:53:08.551 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:53:08.573 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:53:08.573 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:53:08.593 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:53:08.641 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:53:08.642 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:53:08.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:53:08.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:53:08.792 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:53:08.793 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:53:08.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:53:08.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:53:08.856 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:53:08.862 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:53:08.884 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-03-25T19:53:08.911 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-03-25T19:53:08.939 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:53:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:53:08.958 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:53:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:53:08.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:53:09.006 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:53:09.007 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/remote/smithi110 2024-03-25T19:53:09.007 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:53:09.042 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333/remote/smithi178 2024-03-25T19:53:09.042 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:53:09.077 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:53:09.077 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:53:09.079 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:53:09.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:53:09.136 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:53:09.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:53:09.149 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:53:09.149 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:53:09.151 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:53:09.163 INFO:teuthology.orchestra.run.smithi110.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:53 /home/ubuntu/cephtest 2024-03-25T19:53:09.182 INFO:teuthology.orchestra.run.smithi178.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:53 /home/ubuntu/cephtest 2024-03-25T19:53:09.183 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:53:09.192 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:53:09.223 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:53:09.248 DEBUG:teuthology.nuke:shortname: smithi110 2024-03-25T19:53:09.248 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:53:09.274 DEBUG:teuthology.nuke:shortname: smithi178 2024-03-25T19:53:09.275 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:53:09.279 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:27:15.929193', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQnb3SNdRoZlsx73/yAyHFAb4x75q5beKbDacY86ZZwjKwAPqjOPSE/sCU7yAAbXbRFZY8kYDjjaHh33oivqv4='} 2024-03-25T19:53:09.294 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:27:15.927280', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM1L1qYqDlKANLXcEsGewMeo/VqOsIiqfBeFmUTo5wGc3fWIIJIbguckv6PzY9sD8SbC2cqXjydLyGeYjcWgBQQ='} 2024-03-25T19:53:09.329 INFO:teuthology.orchestra.console.smithi110:Power off 2024-03-25T19:53:09.330 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:53:09.346 INFO:teuthology.orchestra.console.smithi178:Power off 2024-03-25T19:53:09.346 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:53:09.358 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2024-03-25T19:53:09.358 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:09.371 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-03-25T19:53:09.372 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:09.381 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-03-25T19:53:09.394 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-25T19:53:13.382 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:13.395 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:13.492 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-03-25T19:53:13.503 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-25T19:53:17.497 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:17.506 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:17.608 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-03-25T19:53:17.615 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-03-25T19:53:17.709 INFO:teuthology.orchestra.console.smithi110:Power off completed 2024-03-25T19:53:17.716 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-03-25T19:53:17.820 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 4583.02040886879 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=baaf4fb94f5d4fcf9ba9d180953680c7 status: fail success: false 2024-03-25T19:53:17.821 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:53:17.933 INFO:teuthology.run:FAIL